How To Understand Microsoft’s Claim About AI Beating Doctors

🔍 Read the full analysis: How To Understand Microsoft’s Claim About AI Beating Doctors on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on office and shipping supplies

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic AI scored higher than generalist physicians on two sets of medical cases in a paper published in Science on October 22, 2026. The study reported 83.2% accuracy on 304 complex cases, compared with 36.1% for doctors using textbooks and the internet and 22.3% without those resources, but it tested historical case records in a simulator, not diagnosis or treatment of real patients.

Microsoft also reported results from 325 emergency department cases: MAI-DxO scored 80.3%, while physicians scored 46.8%. The company’s blog described the system as 1.7 times as accurate as physicians on those cases and more than twice as accurate as doctors using reference materials on the complex cases. Those ratios refer to the study’s measured scores, not to proven performance in hospitals.

The benchmark, called the Sequential Diagnosis Benchmark, begins with a short case summary. A participant must request more information, such as a test result, from a simulated case file. The system is scored on its diagnosis and on estimated testing costs. MAI-DxO uses a language model with a panel of virtual agents that suggest diagnoses, request information and challenge one another over several rounds.

The paper’s results are Microsoft’s own findings. All 15 authors listed in the paper have Microsoft affiliations, including Microsoft AI chief executive Mustafa Suleyman. Publication in a peer-reviewed journal means the paper underwent editorial and scientific review; it does not establish that independent researchers have reproduced the results. The system’s evaluation also used a set of cases selected for difficulty, rather than a representative sample of routine medical visits.

What the Scores Mean for Care

The findings matter because they show how a diagnostic AI might help organize clinical reasoning: it can request information step by step rather than simply choose an answer from a complete case summary. That makes the benchmark more like part of a diagnostic process than a conventional multiple-choice test. But a high score in this setup does not show that the system can safely replace a doctor, improve patient outcomes, or reduce costs in everyday care.

The comparison is also narrower than the phrase “AI beating doctors” may suggest. The physician group consisted of primary care doctors and hospital generalists working individually. The study did not compare MAI-DxO with specialists or teams of clinicians. Since difficult cases may be referred to specialists in practice, the reported gap cannot establish how the AI would perform against the clinicians most likely to handle those cases.

For hospitals and patients, the practical question is not only whether a system can name a diagnosis in a test, but whether it requests appropriate tests, recognizes uncertainty and supports safe decisions in real clinical workflows. Insurers and health systems would also need evidence about costs, staffing, errors and effects on care. The study does not answer those deployment questions.

How Microsoft Tested MAI-DxO

The paper, Sequential diagnosis with language models, appeared in Science, volume 394, pages 423–428. An earlier, non-peer-reviewed version was posted on arXiv in June 2025. The source material says that version used OpenAI’s o3 model and reported an 80% score for AI against 20% for doctors; the later journal paper reports updated results. Figures from different versions should not be treated as a direct measure of progress unless the methods and case sets are also compared.

Details about the benchmark described in the earlier preprint include a separate model that grades diagnoses against the recorded final diagnosis. Physicians reviewing a sample generally agreed with the automated grading, but not perfectly. When a requested test was absent from a case record, a gatekeeper model generated a synthetic result. Test costs were estimated using billing codes and a 2023 U.S. hospital price list, plus a fixed $300 physician visit cost. The authors characterized these costs as first-order estimates. The full Science paper’s details may differ from the preprint.

The reported physician scores also came from a different testing arrangement: doctors were evaluated on a subset of 56 held-out cases, while the AI was evaluated on all 304 complex cases. Microsoft’s published chart data listed physician averages across 764 case attempts without references and 662 with references. This makes the headline percentages useful as results within the study, but not a simple head-to-head trial of doctors and AI encountering identical cases under identical conditions.

““More than twice as accurate as physicians using textbooks and the internet on complex cases.””

— Microsoft, in its blog post, “Building Medical Superintelligence”

Limits Beyond the Simulated Cases

The study did not involve real patients, physical examinations or imaging; it used historical text-based case records. The source material says the complex cases contained no healthy patients and no harmless explanations. That design means the study could not measure how often the system would raise a false alarm when a patient has a common or non-serious condition.

The results also do not establish whether MAI-DxO can be integrated safely into clinical practice, how it compares with specialists or multidisciplinary teams, or whether it improves diagnoses and outcomes. The grading method, synthetic results for unavailable tests and estimated test prices introduce additional limits that readers should keep in mind. Independent replication and performance on a broader range of patients have not been reported in the supplied material.

It is also not clear from the available details how closely the final Science study’s methods match those in the earlier preprint. The published paper is peer-reviewed, but that does not resolve the limits of its test design or show that its reported accuracy will carry over to hospitals.

A Planned Study With Patients

The next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source material, consenting patients are expected to interact with the system alongside their usual care. Such a study could provide evidence about how MAI-DxO behaves in a clinical setting, including how it handles information from actual patient encounters.

The source material does not provide a start date, enrollment target, study protocol or expected results date. Until those details and findings are available, Microsoft’s accuracy figures should be read as performance on a defined simulation, not as proof that the system is ready to diagnose patients independently.

Key Questions

Did Microsoft’s AI diagnose real patients in this study?

No. The study tested historical text-based cases in a simulator. The source material says it did not involve real patients, physical examinations or imaging.

What accuracy did MAI-DxO report?

Microsoft reported 83.2% on 304 complex cases and 80.3% on 325 emergency department cases. These are results from the study’s case sets, not demonstrated accuracy in routine clinical practice.

Were specialists included in the comparison?

The physician comparison described in the source material involved primary care doctors and hospital generalists working alone. Specialists and clinical teams were not included.

Does publication in Science mean the results were independently confirmed?

No. The paper was peer-reviewed, but the source material says the reported results are Microsoft’s own and have not been independently reproduced. Peer review is not the same as replication by another research group.

What is expected to happen next?

A prospective study is planned at Beth Israel Deaconess Medical Center, where consenting patients would interact with MAI-DxO alongside their normal care. The source material does not give the study’s timing or detailed protocol.

Source: ThorstenMeyerAI.com