AI possesses similar accuracy as healthcare professionals in detecting diseases from medical images.

That’s the consensus of researchers at University Hospitals Birmingham NHS in the U.K., who claim to have conducted the first systematic review and meta-analysis of all available evidence from scientific literature on the matter. The group warns, however, that such a finding should be approached with caution, as only a few studies were considered to be up to a standard to be included in the analysis, and that the true potential of deep learning is still uncertain.

"We reviewed over 20,500 articles, but less than one percent of these were sufficiently robust in their design and reporting that independent reviewers had high confidence in their claims,” said professor Alastair Denniston from University Hospitals Birmingham NHS Foundation Trust, U.K., in a statement. “What's more, only 25 studies validated the AI models externally (using medical images from a different population), and just 14 studies actually compared the performance of AI and health professionals using the same test sample.”

Deep learning enables computers to identify patterns in thousands of medical images for greater accuracy and faster diagnoses. Such capabilities have generated enthusiasm around the prospect of deep learning models outperforming humans in diagnostic exams, and led to more than 30 AI algorithms gaining FDA approval in healthcare. Concerns, however, have also arisen around the objectivity of these studies, with many questioning if they are biased in favor of machine learning, and to what degree their findings can be applied in real-world clinical practice.

To address this, Denniston led a systematic review and meta-analysis of 82 studies published between January 2012 and June 2019 that compared the performance of deep learning models and healthcare professionals in detecting diseases from medical imaging. His team also assessed study design, reporting, and clinical value. Of these, data were analyzed for 69 articles considered to have enough information to calculate test performance accurately. The meta-analyses evaluated pooled estimates from 25 articles to validate the findings in an independent subset of images.

Data from 14 studies in the same sample found that, at best, deep learning algorithms correctly detected disease in 87 percent of cases, compared to 86 percent handled by healthcare professionals. They also showed 93 percent specificity in excluding patients who did not have disease, compared to 91 percent of human professionals.

Artificial Intelligence Homepage