A new Boston University (BU) study found that the generative AI model, GPT-4, can answer questions about patient safety with 88% accuracy. The researcher team believes that AI could aid physicians in better recognizing and preventing mistakes in hospitals and clinics.

"We felt it was crucial to investigate the effect of AI on patient safety because the technology is advancing rapidly, particularly with the growth of generative AI and large language models," Dr. Nicholas Cordella, corresponding author and assistant professor of medicine at BU Chobanian & Avedisian School of Medicine, told HCB News. "These innovations are already beginning to shape the field of medicine and will continue to do so, regardless of whether clinical leaders are prepared."

Cordella and his team had GPT-4 take the Certified Professionals in Patient Safety Self-Assessment Exam — a 50-question, standardized multiple-choice certification exam for patient safety professionals. They turned on privacy features so that the answers could not be retained or used to train future models.
The questions were presented without proving the model with an additional training or medical fine-tuning. Its performance was evaluated across five important areas of patient safety; scoring the best in Patient Safety and Solutions, Measuring and Improving Performance, and Systems Thinking and Design/Human Factors, but scoring the lowest in Culture and Leadership.

Although the official threshold to pass the exam is not publicly available, the researchers wrote in the study that they "believe this raw score to be consistent with the performance of proficient human patient safety practitioners." More research is needed, but Cordella believes this study shows that AI has some potential to improve healthcare by helping clinicians tackle preventable harms.

"By studying the current capabilities of AI, we aim to provide insights that can guide its thoughtful and strategic implementation in clinical settings," he said. "Now is the critical time to ensure that we not only maximize the benefits of AI but also carefully address and mitigate the potential risks to patient safety."

A BMJ Quality & Safety research article published in July 2023 found that across all clinical settings, about 795,000 people die or are permanently disabled each year in the U.S. by diagnostic errors.

Cordella outlined two ways that AI can improve patient safety — one being that it can detect and prevent medical errors more effectively. It can do that by identifying risks early and informing targeted interventions.

"Second, AI is poised to transform research, diagnosis, and treatment, opening the door to more accurate and personalized care across various conditions," he added. "However, it’s crucial that patient safety professionals guide AI's implementation to mitigate risks like bias and hallucinations, ensuring its benefits are fully realized without compromising safety."

Artificial Intelligence Homepage