A growing body of research in 2026 is highlighting a persistent gap in medical artificial intelligence: these systems can influence doctors’ decisions and produce impressive diagnostic results without showing that patients ultimately become healthier.

The issue matters to hospitals evaluating AI tools, companies seeking to demonstrate their value, and regulators deciding what evidence should be required after these systems enter clinical practice.

Regulators scrutinize medical AI’s proof problem

The gap between AI’s reported performance and demonstrated benefits for patients is becoming increasingly difficult to overlook. The Financial Times summarized the issue in a headline: “Medical AI has a proof problem.”

The question has moved beyond academic debate. The US Food and Drug Administration is examining many of the same issues as it considers how to assess AI-enabled medical devices before and after deployment. An August 18 discussion paper open for public input until Oct. 19 addresses risk assessment, premarket evaluation and postmarket monitoring.

The FDA emphasized that the document is solely a discussion paper. It is not a draft guideline, a proposed policy change or an indication of future regulatory expectations. Still, the open comment window gives hospitals, device makers and researchers a formal channel to lay out what kinds of evidence they believe clinical deployment should require.

A previous FDA request for public comment on measuring and evaluating the real-world performance of AI-enabled medical devices also raised concerns about model drift and the limitations of relying on static metrics. Model drift — the degradation that can occur when real-world patients or practice patterns diverge from a model’s training data — is one reason a tool validated at launch may behave differently months later. That leaves a broader question: How should safety and effectiveness be measured after an AI system leaves the laboratory and enters clinical care?

AI support did not reduce treatment failures in Kenya

A study published in Nature Medicine on June 26 examined whether an LLM used for clinical decision-making improved patient outcomes. The study involved 103 clinical officers at 16 Penda Health facilities in Nairobi and Kiambu counties. The officers treated patients either with or without assistance from a large language model.

Of the 9,691 registered patients, 9,347 were included in the primary analysis. Treatment failure after 14 days occurred in 2.2% of the AI group and 2% of the control group. The adjusted odds ratio was 0.77, but the difference was not statistically significant (P=0.13). No serious adverse events were associated with the intervention.

The AI-supported group showed better documentation and more appropriate diagnoses and treatment plans. However, those improvements in process measures did not produce an improvement in patient outcomes. That gap is the crux of the proof problem: metrics such as documentation and diagnosis quality are far easier to collect than outcomes like treatment failure, but the Penda results suggest the two can move independently.

A similar pattern appeared in the 2024 RAPIDx AI trial. Among 3,029 patients in the primary analysis, the composite six-month outcome of cardiovascular death, myocardial infarction or unplanned cardiovascular readmission occurred in 26% of patients receiving AI-supported care, compared with 26.4% of those receiving standard care. Within the non-type 1 myocardial infarction group, however, invasive coronary angiography was performed 47% less often with AI-supported care.

Higher test scores have not established real-world benefits

A multicountry randomized trial led by Nicholas Rounding found that GPT-4o improved the performance of 249 doctors on clinical vignettes by 18% in Kenya, 10.7% in Indonesia and 7.2% in the Netherlands (P<0.001). The research, however, used simulated cases rather than real-world clinical situations. Participants in the control group could not use the internet or clinical protocols, and the study did not evaluate harms.

The results show that AI can improve performance in controlled settings, but they do not establish an effect on patient outcomes. That controlled-versus-real-world distinction is precisely the terrain the FDA’s premarket and postmarket questions are aimed at.

Another Nature Medicine study by Li Zhang, Jakob Nikolas Kather and colleagues reported 90.04% accuracy on a seven-disease benchmark. By applying a consistency threshold, the on-site agent retained 49.4% of cases with 98.9% accuracy, while human professionals reviewed the remaining cases. The authors said the findings require further confirmation in trials conducted in real clinical settings. The study is available here.

A doctor in the loop is not enough

An evidence map compiled by Joy Xu, Justin Ko and Joseph Kvedar found that agentic AI is being used not only for administrative work but also for diagnosis, management and other healthcare tasks. As a result, the ability to govern and audit these systems is becoming increasingly important. The evidence map is available here.

A separate commentary in npj Digital Medicine argued that simply having a clinician involved does not guarantee effective supervision or governance. Automation bias can make people more likely to accept a flawed recommendation. Meaningful oversight requires sufficient knowledge, time and authority to challenge the system and intervene when necessary.

Without those conditions, the authors warned, human oversight can become little more than a liability backstop:

Responsibility may collapse onto clinicians expected to catch errors they are not well positioned to detect or correct … a moral crumple zone.

The debate therefore extends beyond whether a human is technically “in the loop.” It also concerns whether that person is equipped to question, override and stop the system when necessary. The commentary is available here.

Commercial stakes increase the importance of evidence

MarketsandMarkets projects that the healthcare AI market will grow from $36.67 billion in 2026 to $194.79 billion by 2031, representing a 39.7% compound annual growth rate. The market forecast comes as hospitals prepare to make more AI purchasing decisions while evidence of patient-outcome improvements remains uneven.

That environment could increase pressure for forms of validation that are more difficult to market but more valuable to providers: prospective testing, local performance monitoring and oversight that can be independently audited. One near-term marker will be the input the FDA receives on its discussion paper, which remains open for public comment until Oct. 19.