AI models have gotten remarkably good at answering medical questions.
Give one a carefully written vignette, a handful of possible diagnoses and a nice clean prompt, and the latest models can look awfully doctor-ish.
Put that same AI in front of months or years of messy medical records, though, and things get considerably more complicated.
Researchers at Carnegie Mellon University have built a new benchmark called Synthetic Hospital designed to test exactly that gap. Instead of asking models isolated medical questions, it gives them longitudinal electronic health records containing multiple visits, lab results, imaging, diagnoses and clinical notes — then asks them to do things closer to actual clinical work. The paper is available on arXiv.
The results are a useful reality check.
Across 10 major AI models, none came close to mastering the benchmark. Models were fairly good at finding relevant information in a chart, but substantially weaker at putting that information together into a diagnosis or useful patient summary. And in one of the study's more interesting findings, wrapping a model inside an AI-agent workflow frequently made performance worse.
That makes Synthetic Hospital less interesting as yet another leaderboard and more interesting as a glimpse at the problem healthcare AI actually has to solve.
Knowing medicine isn't the same thing as navigating medicine.
The problem with testing AI like it's taking an exam
A lot of medical AI research has revolved around benchmarks resembling tests: multiple-choice questions, individual case vignettes or short-answer problems.
Those are useful measures of medical knowledge. They are also very different from what happens inside a hospital.
A real patient may have years of records scattered across clinical notes, labs, imaging reports and past encounters. Some information matters to the current question. Some doesn't. Important evidence might have appeared six visits ago.
The researchers argue that existing public benchmarks rarely capture that longitudinal problem. Real electronic health records would seem like the obvious solution, but they introduce another problem: medical records are heavily restricted by privacy, ethics and data-use requirements. And even when researchers can access them, the chart itself isn't necessarily perfect ground truth. It only records what clinicians happened to document.
Synthetic Hospital tries to get around both problems by creating patients who never existed.
The benchmark contains 1,268 synthetic patients and 5,602 encounters, all assembled from publicly available medical-education material rather than private patient records. Diagnoses, findings and relationships between conditions are mapped onto medical standards including ICD-10-CM, SNOMED CT and LOINC.
That gives researchers something unusually valuable: a patient chart that looks realistic, but where the people building the benchmark actually know what the complete underlying patient state is supposed to be.
The project's code and data are publicly available on GitHub.
Doctors couldn't reliably tell the fake charts from the real ones
Synthetic medical data isn't particularly useful if it screams hello fellow physicians, I was generated by a computer.
So the researchers tested that too.
Ten licensed physicians reviewed a mixture of real and synthetic patient records without being told which were which. Across 100 judgments, the physicians correctly identified real versus synthetic records 53% of the time — statistically indistinguishable from the 50% chance baseline.
That doesn't mean Synthetic Hospital perfectly reproduces an actual hospital population. The authors explicitly say it doesn't. Its cases come from medical-education materials, which means its disease mix and patient distribution aren't intended to mirror real-world epidemiology.
But individual charts apparently looked plausible enough to fool physicians.
And that makes what happened when the models entered the hospital much more interesting.
AI can find the evidence. Understanding the whole patient is harder.
The researchers tested models including GPT 5.3, Gemini 3.1, Opus 4.6, Kimi 2.5 Thinking, DeepSeek V3.2, GLM 5, Qwen 3.5, Mistral Large, Llama 4 Scout and Gemma 3.
There wasn't one model that simply dominated everything.
On reconstructing a patient's diagnoses across their longitudinal history, the strongest result was Kimi 2.5 Thinking with a severity-weighted F1 score of 0.732. GPT 5.3 followed at 0.703. The smallest model tested, Gemma 3 27B, landed at 0.287.
The study also included seven physicians on a smaller, matched 13-patient subset. Their average diagnosis score was 0.664, with individual results ranging widely from 0.21 to 0.89. That's worth treating cautiously: it's a small subset and a specific benchmark metric, not evidence that any AI model is broadly "better than doctors."
The clearer pattern appears when you compare types of tasks.
Every model scored above 0.74 on retrieving relevant chart evidence. But the best whole-patient summarization scores hovered around 0.5, and imaging-indication performance remained in roughly the same neighborhood.
In other words: AI is getting much better at finding the right page.
Making sense of the whole book is another matter.
That's especially important as healthcare companies increasingly explore agents capable of operating over medical records, something we also see in the broader shift toward AI agents navigating complex healthcare workflows.
Because Synthetic Hospital found that adding more "agent" doesn't automatically solve the problem.
The agent actually made things worse
The researchers took three models — GPT 5.3, Mistral Large and Llama 4 Scout — and compared several configurations.
One received all the necessary chart context directly.
Another was turned into an agent operating through a 13-tool, Epic-style healthcare API.
A third agent had to retrieve the relevant information itself.
You might expect the tool-using version to be the sophisticated one.
Usually, it wasn't.
When models already had the relevant chart information, putting them into a multi-step agent loop reduced performance across nearly every model-task combination outside patient diagnosis. Summarization and evidence retrieval dropped by as much as roughly 0.19, while Llama 4 Scout's imaging score fell by about 0.24.
Failed agent runs were expensive too: they consumed an average of about 145,000 input tokens, compared with around 60,000 for successful runs.
Then comes the twist.
For longitudinal diagnosis — where finding evidence scattered throughout the chart is actually part of the problem — self-retrieving agents helped all three models. Compared with the same models receiving full context directly, the self-retrieval setup improved severity-weighted diagnosis F1 by roughly 0.14 to 0.34.
That suggests an important distinction we're going to encounter far beyond medicine:
Agents aren't inherently better than models. They're better when agency matches the shape of the work.
If the necessary information is already sitting in the prompt, adding a planning loop, tool calls and repeated model interactions may just introduce more opportunities to lose the plot.
If the job requires exploring an environment and deciding what evidence to gather, agency can suddenly become valuable.
It's similar to what we've seen elsewhere with research showing that the harness around an AI model can matter nearly as much as the model itself.
More machinery isn't automatically better machinery.
The next healthcare benchmark may matter as much as the next healthcare model
Synthetic Hospital is still an experiment.
Its physician realism study was relatively small. The synthetic charts don't intentionally recreate many of the contradictions, missing information and documentation errors found in real EHRs. Its agent evaluation covered only three models, 100 patients and one agent scaffold.
So nobody should read this as proof that we've figured out how to simulate a hospital perfectly.
But the paper points toward something important.
For years, medical AI progress has often been measured by asking whether models know enough medicine.
We're reaching the point where that question isn't sufficient.
The harder problems look much more like actual knowledge work: retrieving information from enormous histories, distinguishing signal from noise, maintaining context across multiple steps and knowing when not to add complexity.
Synthetic Hospital gives researchers a way to study those problems without handing them real patient records.
And its first lesson is a useful one.
AI may already know a surprising amount about medicine.
Getting it to reliably understand a patient is still a very different challenge.
