Every medical school we have spoken to this year has been pitched an AI virtual patient. The pitch is almost always the same — a chatbot that plays a patient, available at any hour, for a fraction of what a standardized patient costs.
The pitch is not wrong. But the published evidence on these tools is now large enough to be useful, and it points at a specific set of questions that a product demo will not answer on its own. This is what the research actually found, and what we would check before signing anything.
What the evidence actually shows
The most cited study in this space is Holderried et al., published in JMIR Medical Education in January 2024 (doi:10.2196/53961). Twenty-eight medical students practised history taking against a GPT-3.5 chatbot primed with an illness script. The researchers recorded 826 question-answer pairs, a median of 27.5 per conversation, and had them assessed for plausibility.
Two results matter for a buyer.
The first is encouraging: of 860 pairs assessed, 842 (97.9%) were rated plausible. Students found it usable, returning a Chatbot Usability Questionnaire score of 77 out of 100. Something real is working here.
The second is the one that does not appear on slides.
Off-script questions get invented answers
When students asked about something the case script did not cover — 195 questions — the model answered with fictitious information in 110 of them. That is 56.4%.
Sit with what that means in a teaching context. A student asks a reasonable clinical question. The author of the case never wrote an answer for it. The model, built to continue a conversation rather than to refuse one, produces a confident, plausible-sounding symptom that is not part of the case. The student incorporates it, reasons from it, and may then be marked against a presentation that never existed.
This is not an argument against AI virtual patients. It is an argument for asking one specific question in every demo: what happens when a student goes off-script? There are three possible designs — the system refuses, it deflects back to the script, or it generates. Only the third is dangerous, and it is also the cheapest to build, which is why it is common.
A system that grounds every answer in an authored case has a ceiling on what it can say. That ceiling is the feature.
The second gap: most of them stop at the history
In the same 2024 study, 94.7% of what students asked was history taking. That reflects what the tool could do, not what a clinical encounter is.
Look at what an OSCE station actually assesses. A student takes a history, yes — then examines the patient, decides which investigations are worth ordering, interprets what comes back, and commits to a diagnosis and a plan. The history is the opening move.
Almost every AI virtual patient on the market today is a text chatbot that ends where the history ends. There is nothing to examine, because there is no body. There are no investigations to order, because there is no result set behind the case. The student never has to commit to anything, because the conversation simply stops.
That is a legitimate product — history taking is a skill worth drilling, and drilling it cheaply at volume has value. But it is a fraction of the encounter, and it should be priced and timetabled as a fraction, not as a simulation of clinical practice.
When we built MedSim, this was the decision that shaped everything else: the patient is a 3D character you can actually examine, with visible clinical signs, and the encounter runs through investigations to a diagnosis you have to commit to. It costs far more to build than a chat window. It is also the only version that assesses what an OSCE assesses.
Five things to check in a demo
1. Is the case bounded, or is the model improvising?
Ask the vendor to have someone ask the patient about a symptom that is not in the case. Watch what happens. If the answer is fluent and specific, ask where it came from. The 56.4% figure above is what unbounded generation looks like when someone measures it.
2. Does the encounter go past the history?
Can the student examine the patient and find something? Order a test and get a result that belongs to this case? Commit to a diagnosis and be wrong in an informative way? If the product stops after the conversation, you are buying history-taking practice, which is worth having — at history-taking prices.
3. Is it spoken, or is it typed?
The authors of the 2024 study flag this themselves as a limitation: their interaction was written, "whereas history taking is usually spoken." Typing "do you have chest pain" is not the same skill as asking it, hearing a hesitant answer, and following it. If a platform is text-only, it is training a related skill, not the clinical one.
4. What exactly does it mark?
Ask to see a real score sheet from a real attempt, not a screenshot from the website. The questions that matter: does each mark cite the evidence behind it, so a student can see why? Does it assess the reasoning or only the final answer? And can a faculty member override a mark they disagree with? A tool that cannot be overridden will eventually be ignored, because the first time it marks a good student wrongly, the faculty stops trusting it.
5. Whose language does the patient speak?
Patients do not present in textbook English. If your students will practise in one language and qualify in another, or take histories from patients who use dialect and idiom, a platform that only speaks formal English is teaching half the task. This is not a localisation checkbox — a patient who says the regionally normal thing for their symptom is a different teaching object from one who recites a translated script.
What the evidence does not show yet
Be careful of the claim that these tools improve clinical skill, because nobody has demonstrated it convincingly.
The Cardiff University pilot on SimFlow.AI is the most positive published result we have found: self-reported confidence in history taking and risk assessment rose significantly (z = −3.750, P < .001), 94% of participants said the platform helped them identify strengths and weaknesses, and 94% rated the AI responses realistic and clinically plausible (doi:10.1192/bjo.2026.11161). It is a genuinely good signal.
It is also 20 students who completed all components, measuring self-reported confidence, in a pilot service evaluation. The authors themselves conclude the platform should complement rather than replace patient actors.
Confidence is not competence. A student can feel significantly more prepared and perform identically. Until somebody publishes OSCE outcomes against a control group, the honest claim for this entire category is: students use it, students like it, students report feeling more prepared, and the effect on marks is unmeasured.
We would rather say that than quote a number we cannot stand behind. If a vendor quotes you one, ask for the paper.
Where this leaves a curriculum lead
The economics are real. Standardized patients are paid by the hour — posted US program rates in our research ran from roughly $15–$17.50 at Virginia Tech Carilion to $25–$32 at UNBC, before the scheduling, training and room overhead that never appears in the hourly figure. That is why a student typically gets one or two encounters on a topic rather than twenty.
Software does not have a calendar. The realistic role for an AI virtual patient is not to replace the person in the room; it is to make sure that when the student reaches the person in the room, they have already made their obvious mistakes somewhere cheaper.
That only works if the practice resembles the assessment. A chatbot that stops at the history, types instead of speaks, invents answers when pushed, and reports a verdict with no reasoning will produce confident students who have rehearsed the wrong thing.
Ask the five questions. Make the vendor demo the failure cases, not the happy path.
We build MedSim — virtual patients for medical schools, with spoken consultations in English, Egyptian Arabic and Saudi Arabic, physical examination with visible signs, investigations, and a score sheet that quotes the student back. If you are evaluating platforms and want a blunt conversation about where ours does not fit, we are straightforward to reach.