Artificial intelligence is rapidly reshaping healthcare, from sepsis prediction tools to drug development pipelines. According to FDA Commissioner Robert Califf, MD, this evolution is unprecedented in scope, sparking both optimism and serious regulatory challenges. In a recent interview with JAMA+ AI Editor-in-Chief Roy Perlis, MD, Califf emphasized the FDA’s role in ensuring AI’s safe and effective use, likening it to monitoring an ICU patient—an ever-changing system requiring continuous oversight.
The New Frontier: Postmarket Monitoring of AI Tools
The FDA has approved nearly 1,000 AI-driven devices, but Califf highlighted that AI is not a “build it and leave it” technology. Unlike traditional drugs or devices, AI algorithms evolve in real time, influenced by environmental factors and new data inputs. For instance, a sepsis prediction algorithm widely used in health systems faced scrutiny after its real-world performance diverged from initial trial results. To address such risks, Califf advocated for robust postmarket monitoring, where algorithms are assessed continuously to ensure they perform as intended. He suggested this requires shared accountability among regulators, developers, and health systems.
Why Accountability in AI Regulation Matters
For clinicians and health systems, the implications of unregulated or poorly monitored AI tools are significant. Erroneous algorithms can misguide care decisions, exacerbate health disparities, and diminish trust in emerging technologies. Califf pointed to the need for clinical and professional organizations to push for self-regulation, ensuring AI aligns with patient safety and equitable care delivery. He also expressed concern that many current AI applications focus on optimizing health system finances, which often risks prioritizing financial incentives over equitable health outcomes.
He noted that while financial considerations are a necessary part of operating a health system, it is essential to ensure they are balanced with accountability for improving patient outcomes. Without this balance, algorithms could unintentionally deepen disparities or undermine trust in their use.
Building Expertise and Ecosystems for Future Regulation
As AI evolves exponentially, the FDA faces the challenge of keeping pace with the technology. Califf highlighted that the agency is attracting talent capable of navigating these complexities, but acknowledged ongoing challenges in regulating disruptive technologies like generative AI and large language models. These tools, while transformative, often lack transparency and pose risks such as generating inaccurate or biased outputs.
Looking ahead, Califf emphasized the need for stronger partnerships with health systems and the broader clinical community to monitor AI tools in real-world settings.
A Call for System-Wide Responsibility
AI’s potential to revolutionize medicine comes with a responsibility to ensure safety, efficacy, and equity. While the FDA continues to set the regulatory framework, Califf stressed the importance of shared accountability across all stakeholders, including clinicians, health systems, and policymakers. Postmarket monitoring, he argued, is not solely the FDA’s responsibility but a collective effort requiring vigilance from every level of the ecosystem.
In his discussion, Califf underscored that the challenges of overseeing AI tools extend beyond any single organization or regulatory body. Addressing these issues effectively will require collaborative efforts from the entire healthcare ecosystem, he noted, emphasizing the need to work together to safeguard AI’s transformative potential.
