A Measured Look At Microsoft’s AI-versus-Doctors Results

🔍 Read the full analysis: A Measured Look At Microsoft’s AI-versus-Doctors Results on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on monitors, keyboards and dev gear

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s MAI-DxO diagnostic AI scored higher than generalist physicians on difficult medical cases in a study published in Science on October 22, reporting 83.2% accuracy on 304 case records compared with 36.1% for doctors who could use textbooks and the internet. The results come from a text-based benchmark using historical cases, not from treating patients, and do not show whether the system improves real-world diagnosis or outcomes.

The paper, Sequential diagnosis with language models, describes the Sequential Diagnosis Benchmark, in which each participant begins with a short case summary and requests further information, such as test results, from a simulated gatekeeper. The benchmark evaluates both the final diagnosis and the cost of tests requested along the way. Microsoft says MAI-DxO reached 83.2% accuracy on 304 complex cases drawn from the New England Journal of Medicine. Physicians scored 36.1% with reference materials and 22.3% without them.

On a separate set of 325 emergency department cases, Microsoft reports an accuracy score of 80.3% for the system, compared with 46.8% for physicians. The paper’s authors are all affiliated with Microsoft, including Microsoft AI chief executive Mustafa Suleyman. The findings have been peer-reviewed and published, but the results reported in the paper remain Microsoft’s study results; the source material provides no independent replication.

MAI-DxO coordinates a language model through several rounds of proposed diagnoses, questions and tests, with virtual agents challenging one another. Microsoft’s chart also reports that GPT-5 alone scored 78.9% on the complex cases, before the MAI-DxO orchestration layer. That comparison suggests most of the reported diagnostic score came from the underlying model, while the additional system reportedly added about four percentage points and reduced estimated test costs.

What the Scores Mean for Care

The findings matter because medical AI is often judged on questions where all relevant information is already supplied. This benchmark instead attempts to measure a more practical task: what information to request, which tests to order and when to settle on a diagnosis. A system that can gather relevant evidence efficiently could eventually be useful to clinicians, particularly when cases are complex.

But a higher score on this benchmark does not establish that MAI-DxO is safer or more effective than doctors in clinical practice. The comparison was between a multi-agent AI system and individual generalists working alone, not specialist physicians or clinical teams. The cases were selected as difficult teaching examples, and the benchmark did not include healthy patients or benign explanations. It therefore could not measure how often the system would raise false alarms in a broader patient population.

That distinction matters to patients, hospitals and insurers. The study supports further investigation of the system’s performance on a particular kind of case; it does not establish that patients should rely on it, that hospitals should replace clinicians, or that its estimated test savings would carry over to routine care. The cost figures are modelled estimates, not evidence of savings achieved in a hospital.

How Microsoft Tested MAI-DxO

The authors designed the benchmark to imitate the step-by-step process of diagnosis. Each case starts with a short presentation, and participants must request details from a simulated source holding the full record. As described in the earlier 2025 preprint, a model evaluates diagnoses against the published answer, and it can generate synthetic results when a requested test is absent from the case file. Test costs are estimates based on billing codes and a 2023 U.S. hospital price list, with a fixed $300 physician visit charge. The source material says these methods are described in the preprint because the full Science article is behind a paywall; details in the final paper may differ.

The preprint described 21 U.S. and UK generalists: 17 in primary care and four hospital generalists, with a median of 12 years’ experience. It said doctors completed a held-out subset of 56 cases, while the AI was evaluated on all 304. The Science paper reports the final comparison figures, including an open-book physician group. These are not like-for-like conditions in every respect, and doctors were not tested as specialist teams.

An earlier, non-peer-reviewed version appeared on arXiv in June 2025. That preprint reported a more dramatic gap, comparing an earlier AI system based on OpenAI’s o3 with doctors. Microsoft’s October 2026 blog post presents the newer results in ratio form, while the Science paper supplies the measured percentages. The published study should be distinguished from both the earlier preprint and the company’s promotional description.

“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”

— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch

Limits of the Evidence So Far

It is not yet known how MAI-DxO would perform on prospective, real-world patients, where symptoms may be incomplete, tests can be unavailable and diagnoses can change. The benchmark used historical text records rather than physical examinations, imaging interpreted in a clinical workflow or care delivered alongside a physician. No patient outcomes were measured in the reported test.

The available source material also does not establish whether an independent team has reproduced the findings. Peer review evaluates the paper’s methods and reporting, but it is not the same as external replication. Some procedural detail cited here comes from the 2025 preprint, and the source notes that the final Science version may differ. The effect of the AI’s higher accuracy on missed diagnoses, unnecessary testing, false alarms and total costs remains unclear.

Testing the System With Patients

The next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source material, consenting patients are expected to interact with the system alongside their usual care. That design could provide evidence about how the tool works in a clinical setting while patients continue to receive standard care.

Details such as the study’s start date, size, evaluation measures and publication schedule are not provided. Until results from that work or other independent evaluations are available, the Science paper is best read as evidence about MAI-DxO’s performance on a defined set of simulated diagnostic cases—not as proof that it can replace doctors or improve patient outcomes.

Key Questions

What did Microsoft’s study find?

Microsoft reported that MAI-DxO achieved 83.2% diagnostic accuracy on 304 complex case records, compared with 36.1% for generalist physicians using textbooks and the internet. The comparison was part of a text-based benchmark, not a trial of patient care.

Was the study peer-reviewed?

Yes. The paper, Sequential diagnosis with language models, was published in Science on October 22, 2026. Peer review does not mean the findings have been independently reproduced.

Did the test involve real patients?

No. The reported benchmark used historical case records in a simulator. It did not test MAI-DxO on patients receiving care, and it did not measure patient outcomes.

Did MAI-DxO outperform specialist teams?

The reported comparison was with generalist physicians working individually. The source material says specialists and physician teams were not included, so the study does not answer how the system compares with them.

What happens next?

A prospective study is planned at Beth Israel Deaconess Medical Center, where consenting patients would interact with the system alongside their usual care. The source material does not give a start date or further study details.

Source: ThorstenMeyerAI.com