Researchers outline guidance for safer AI-enabled medical devices
A publication in NEJM AI outlined seven human factors-related risks for AI-enabled medical devices and called for clearer workflows, training, fallback options and post-market monitoring. The analysis said current approval requirements only partially address these risks, while separate evidence cited persistent accuracy and verification problems in medical AI use.
A recent publication in NEJM AI said the safety and performance of AI-enabled medical devices not only depends on algorithms or technical specifications, but also on how people use these devices and applications. The authors said existing regulatory requirements for approval have so far only partially addressed many human factors-related risks, creating gaps that can impact the safety and quality of care.
The analysis identified seven key risks and developed practical recommendations for manufacturers and regulatory evaluators that can be integrated into existing regulatory and documentation processes. The risks include outputs being misunderstood or misinterpreted because of the sometimes-opaque nature of AI systems, miscalibrated trust that leads users to rely too heavily on AI assistance or ignore relevant recommendations, automation bias, potential deskilling, technostress among users, indication creep, and errors related to system changes or different operating modes.
The framework recommends developing and deploying AI-based medical devices in a way that clearly defines the users, in which context the systems are applied, and which tasks are assigned to humans and which to the system. It also said results should be presented in a way that is easy to understand, integrated into existing clinical workflows, and supplemented by training where needed as well as safe fallback options in the event of system failures.
The authors emphasized the importance of continuous monitoring after market entry. Usage patterns, potential misuse, or overreliance on AI systems should be systematically observed and corrected as needed, and changes to the systems must also be communicated transparently so that work processes can be adjusted accordingly.
The broader risks of unreliable medical AI are already measurable. In a 2025 cross-industry survey, 44% of organizations reported experiencing negative consequences from generative AI use, with average financial losses of $4.4 million per incident. A 2023 JAMA Network Open paper about AI-generated discharge summaries demonstrated that 18% of cases contained incomplete or misleading information, while research cited in the same discussion found that only 39% of individuals verify AI-generated information using external sources.
The analysis said such factors can create additional burdens or unexpected failures in clinical practice even when the technical performance of a system itself is strong. The recommendations were deliberately formulated in general but regulatory-aligned terms so that they can be applied to different AI-enabled medical devices and application scenarios, with further testing planned through concrete pilot applications.