While promising, Dr. Xiaoxuan Liu, another author from the University of Birmingham, U.K., cautions that the AI algorithms did not “substantially outperform” humans. He adds that several limitations within the methodology and reporting of AI-diagnostic studies, including his analysis, must be addressed. These include the fact that he and his colleagues assessed deep learning in isolation in a way that does not reflect clinical practice; the fact that few prospective studies were completed in real clinical environments; the need for high-quality comparisons in patients to determine diagnostic accuracy as well as datasets; and the elimination of poor reporting, which is common and caused by most studies not reporting missing data.

"There is an inherent tension between the desire to use new, potentially lifesaving diagnostics and the imperative to develop high-quality evidence in a way that can benefit patients and health systems in clinical practice," he said in a statement. "A key lesson from our work is that in AI — as with any other part of healthcare — good study design matters. Without it, you can easily introduce bias, which skews your results. These biases can lead to exaggerated claims of good performance for AI tools which do not translate into the real world.”

The findings were published in The Lancet Digital Health.

Back to HCB News

Artificial Intelligence Homepage