Microsoft reported a strong result on a difficult diagnostic benchmark in June 2025. That result concerned an AI research system working through published case histories, not a clinical trial showing that AI diagnoses patients better than doctors in ordinary practice. This article preserves the historical announcement and explains that distinction. Related coverage: healthcare.
What Microsoft Announced in June 2025
Microsoft described its goal as “medical superintelligence.” Its June 30 announcement, written by Dominic King and Harsha Nori, presented the Microsoft AI Diagnostic Orchestrator (MAI-DxO) and the Sequential Diagnosis Benchmark (SD Bench). The phrase described a research direction; it did not establish an approved clinical product.
The best reported configuration, MAI-DxO using OpenAI’s o3, reached 85.5% diagnostic accuracy. Microsoft reported approximately 20% average accuracy for the participating physicians on the cases they completed under the study’s restricted conditions. Those percentages describe this benchmark comparison, not general medical competence.
The result warrants further investigation. A higher score on published case histories does not demonstrate faster treatment, fewer deaths or better outcomes for real patients. Those questions require clinical evidence.
A Sequential Benchmark of Complex Case Histories
Microsoft contrasted its benchmark with medical multiple-choice evaluations. SD Bench required a system to request information and tests before making a diagnosis. That design adds a decision sequence; it does not by itself resolve how well a model will perform with real patients.
SD Bench used 304 complex cases from recent New England Journal of Medicine case records, according to Microsoft. These were unusually difficult diagnostic cases rather than a representative sample of everyday consultations. Related reporting: New England Journal of Medicine.
In the benchmark, the system received limited initial information, asked questions, selected simulated diagnostic tests and revised its working diagnosis. The test results came from the benchmark’s case information. This was a structured research exercise, not patient care delivered by an autonomous doctor.
What the Virtual Panel Means
MAI-DxO organized a language model into a virtual panel with different diagnostic roles. Microsoft tested this approach with separate foundation-model configurations, including GPT, Llama, Claude, Gemini, Grok and DeepSeek. The announcement does not establish that the reported best result came from a joint panel combining those providers’ models.
The roles let the system propose possibilities, challenge reasoning and consider which tests to request. They were simulated roles inside the orchestration process, not licensed specialists examining a patient. The benchmark does not support a claim that no human doctor can match the system’s expertise.
Microsoft reported improved accuracy across the model configurations it tested with orchestration. Its cost comparisons concerned the benchmark’s diagnostic-test choices. Related historical coverage: Microsoft’s research.
Diagnostic-Test Costs in the Experiment
A useful diagnostic system would need to consider the cost of the tests it requests as well as its answers. SD Bench attached costs to diagnostic tests so that Microsoft could compare accuracy and expenditure within the experiment.
Microsoft reported that MAI-DxO could reach a better accuracy-and-test-cost tradeoff than the comparison configurations. The primary announcement does not support this article’s earlier blanket claim of a 20% reduction in healthcare costs. That claim has been withdrawn.
Benchmark test costs are not a complete account of running a healthcare service. The study did not establish actual savings for patients or providers after deployment, or measure every cost and consequence of using the system in practice.
The Physician Comparison and Its Limits
Microsoft recruited 21 practicing physicians from the United States and United Kingdom, each with between five and twenty years of experience. The physicians could not consult colleagues, textbooks or generative AI during the benchmark. Microsoft acknowledged that this differs from a normal working environment.
The restricted comparison matters when interpreting the result. Clinicians ordinarily have access to additional resources and colleagues; a score under these experimental conditions should not be presented as the performance of a clinical team in practice. Related historical reporting: Newsweek.
The 85.5% figure belongs to the best reported MAI-DxO-plus-o3 configuration. It should not be generalized to every model, or described as over 80% accuracy in most configurations without a matching result. The approximately 20% physician average also remains specific to this experiment.
A Research System Is Not a Replacement for Care
The June announcement did not demonstrate that AI could replace physicians. Diagnosis is only one part of care, and the benchmark did not evaluate an autonomous system’s responsibility for treatment or a patient’s complete clinical course.
Reading a written case and selecting its diagnosis does not establish that a system can safely examine patients, communicate uncertainty, prescribe treatment or manage follow-up. Those capabilities should not be inferred from the reported diagnostic score.
Dominic King and Harsha Nori framed the work as a step toward further research. Any role in clinical care requires evidence beyond this announcement. Related historical reporting: Dr. Dominic King.
What Had Not Been Validated
Microsoft explicitly identified the complex NEJM case selection as a limitation. Routine presentations are different from the difficult diagnostic puzzles in SD Bench. The announcement called for tests with more common symptoms before drawing broader conclusions.
It also said that moving toward clinical practice would require rigorous real-world testing, appropriate governance and attention to regulatory requirements. At the time of the announcement, Microsoft described the benchmark and orchestrator as research demonstrations, not publicly available clinical tools.
Microsoft said it intended to expand testing with healthcare organizations and external experts. That stated direction is not evidence that a particular clinical trial subsequently began or succeeded. Related June 2025 coverage: partnering with health organizations.
Health-Related Sessions Are a Separate Measure
Microsoft said Bing and Copilot generated more than 50 million health-related sessions per day. A session is not necessarily a unique patient, a diagnosed case or a single query. That usage statement was separate from MAI-DxO’s research evaluation.
The announcement did not show that MAI-DxO was handling those sessions or diagnosing their users. A consumer AI conversation should not be confused with a validated diagnostic service.
Better diagnostic support could be valuable, but this study did not measure improved access to specialist care or outcomes in rural settings. Those remain possible applications to investigate rather than demonstrated benefits.
What Would Be Needed for Clinical Use
Microsoft’s own qualifications point to further clinical evaluation, safety work and governance before a system like this could be used in patient care. Benchmark performance alone does not establish regulatory authorization or safe deployment.
The announcement did not provide a public launch date, patient-facing product or verified commercial availability for MAI-DxO. This historical report makes no claim about a later product release or current clinical access.
Microsoft said it was working toward external review of the research. A plan to submit work for peer review is different from completed independent peer review. Related reporting: Suleyman indicated.
Questions for Medical Training
The result raises questions about how clinicians might assess AI-generated diagnostic suggestions. It does not establish that medical training should abandon diagnostic reasoning or change specialty requirements.
A future evaluation could examine how clinicians use such suggestions, detect mistakes and communicate uncertainty. SD Bench did not measure whether that collaboration improves treatment or makes training more effective.
Likewise, the experiment did not demonstrate a change in specialists’ roles. Claims about the future organization of medical practice should remain separate from the measured diagnostic result.
Limits on Global Applications
The announcement did not establish that its results apply across countries, healthcare systems or patient populations. Its physician comparison involved participants from the United States and United Kingdom, and its cases came from NEJM records.
Improved diagnoses in rural hospitals, reduced medical travel and better local outcomes were not measured. This report withdraws earlier language that treated those possible benefits as outcomes of the study.
Any wider use would need evidence about the intended patients and setting. The June 2025 research announcement alone cannot answer those deployment questions.
How to Read the Result
The strongest supported claim is specific: Microsoft reported 85.5% diagnostic accuracy for MAI-DxO with o3 on its complex sequential benchmark. The experiment provides a reason to study the approach, with substantial limits on what can be inferred about clinical practice.
Microsoft called for further real-world testing. This article has not verified later clinical outcomes, a current deployment, regulatory clearance or an independent replication of the result.
We did not run MAI-DxO, reproduce SD Bench or assess patients. The figures above are attributed to Microsoft’s dated research announcement. They are not results from testing by Kingy.ai.
For readers comparing diagnostic AI claims, keep three questions separate: what was measured, under which conditions, and whether patient outcomes were evaluated. Here, the supported evidence concerns a difficult case-history benchmark, not a demonstrated replacement for medical care.
Sources
- Entrepreneur – Microsoft Claims Its AI Is Better Than Doctors at Diagnosing Patients
- Heise Online – On the way to medical superintelligence: Microsoft presents research
- Rapamycin News – Microsoft Says Its New AI System Diagnosed Patients 4 Times More Accurately
- Repertoire Magazine – Microsoft takes step toward ‘medical superintelligence’
- Microsoft AI – The Path to Medical Superintelligence
- Newsweek – New Microsoft AI Research Edges Towards ‘Medical Superintelligence’
- AI Magazine – How Microsoft’s AI Sets New Standards for Medical Diagnosis
- GeekWire – AI vs. MDs: Microsoft tool hits 85.5% accuracy in tough diagnoses
The Kingy Brief
Get the free AI price sheet.
26 AI plans and models on one page, from official pricing pages, updated every month. Free when you join The Kingy Brief: AI changes, original tests and one practical thing to try, Fridays at 09:00 Vancouver time.
Free · Double opt-in · Unsubscribe anytime
Sign up for The Kingy Brief on Beehiiv.
Signup help and newsletter schedule
Signup form provided by Beehiiv. After submitting, check your inbox for "Confirm your subscription to The Kingy Brief" and open its confirmation link. Your price sheet download is on the page that opens. Check Spam or Promotions if you cannot find it.
Fridays at 09:00 Vancouver time: source-checked AI changes, original tests and one practical thing to try. We skip a week when there is not enough verified material. Free. Unsubscribe anytime.