By Angela Anderson

The integration of artificial intelligence (AI) into clinical decision support (CDS) systems holds immense promise for clinical practice. It can help clinicians analyze vast amounts of information, reduce manual work, and deliver key insights at the point of care. As promising as these advancements are, they come with potential risks in clinical workflows.

Algorithm vigilance, also known by the tongue-twisting name algorithmovigilance, refers to the continuous monitoring and evaluation of algorithms to ensure they maintain reliability, safety, and effectiveness over time. Much like the practice of pharmacovigilance—which monitors drugs for adverse effects after they hit the market—algorithm vigilance ensures that AI tools perform as intended in real-world clinical environments. Ongoing attention and oversight matters as performance – specifically accuracy – can wane or drift over time. Probabilistic and non-deterministic models are especially vulnerable to these issues, but they are also present in the algorithms that underpin large language models and agents becoming widely adopted today.
Using algorithm vigilance to mitigate risks of AI
AI has proven its ability to transform health care by improving efficiency, reducing cognitive load, and enhancing patient outcomes. Predictive models can assist in early diagnosis and resource allocation. AI can streamline clinical documentation, and decision-support tools can help reduce medication errors.

However, as with any innovation in medicine, we must also recognize the risks. One significant risk is bias. AI can be tainted by systemic biases reflected in health care data, which may lead to disparities in patient care. For example, algorithms trained on datasets without appropriate representation might deliver less accurate predictive outcomes for minority groups, people with rare clinical conditions, and other under-researched populations. Securing access to quality data with appropriate representation is no easy task.

Another risk is the lack of trust many clinicians have when it comes to AI. Some health care providers are hesitant to rely on AI tools, because they want to be sure they are using evidence-based, accurate information to diagnose and treat patients. Effective education, transparent design, proactive governance, and assurance of continuing evaluation can mitigate these concerns.

Monitoring the performance of all AI agents and tools is important for the duration they are used in clinical practice. AI performance may change over time and as populations, treatments, and standards of care evolve. This phenomenon, known as algorithmic drift, further emphasizes the need for ongoing oversight.

Artificial Intelligence Homepage