Study: AI chatbots can run with medical misinformation, stronger safeguards needed

Artificial intelligence has already found its way into exam rooms, labs, and even bedside conversations. But a new study out of the Icahn School of Medicine at Mount Sinai throws cold water on any rush to let chatbots roam free in health care. The research shows that popular AI chatbots can be easily misled into parroting — and even embellishing — false medical details. The findings spotlight a pressing need for stronger guardrails before these systems can be considered trustworthy companions for patients and clinicians alike.

As lead author Mahmud Omar, MD, put it: "They not only repeated the misinformation but often expanded on it, offering confident explanations for non-existent conditions."

How did it work?

The researchers devised a clever stress test: fictional patient cases sprinkled with fabricated medical terms — completely made-up diseases, symptoms, or tests. They then asked top large language models to review the scenarios.

• Round one: The chatbots received the scenarios straight, no hints attached.
• Round two: The same cases were presented again, but with a simple one-line reminder: the information might be inaccurate.

Without that nudge, the chatbots didn't just accept the bogus details — they doubled down, spinning out elaborate and convincing medical-sounding explanations. But once the caution line was added, the hallucinations dropped sharply. As co-author Eyal Klang, MD, observed: "Even a single made-up term could trigger a detailed, decisive response based entirely on fiction. But... a well-timed safety reminder cut those errors nearly in half."

Why does it matter?

The stakes are obvious. Misinformation in medicine isn't just a nuisance — it can be life-threatening. If an AI system confidently explains a fake disease or prescribes an imaginary treatment, the consequences could be severe. The study offers a sobering glimpse into how fragile trust in these tools can be.

And yet, the researchers didn't stop at diagnosis — they suggested a remedy. The discovery that a tiny tweak in prompt design can dramatically improve safety points toward practical, near-term solutions. As Omar noted, "Small safeguards can make a big difference."

For developers, hospitals, and regulators, this could be a simple but powerful way to stress-test AI systems before they touch real patients.

The context

The paper, published in Communications Medicine, arrives at a moment when AI is rapidly infiltrating clinical settings. Doctors lean on chatbots for quick summaries. Patients query them about symptoms late at night. And health systems, eager for efficiency, are exploring how to integrate them into decision-making.

But the Mount Sinai team warns against complacency. Girish N. Nadkarni, MD, MPH, who heads Mount Sinai's Department of Artificial Intelligence and Human Health, framed it bluntly: "A single misleading phrase can prompt a confident yet entirely wrong answer. The solution isn't to abandon AI in medicine, but to engineer tools that can spot dubious input, respond with caution, and ensure human oversight remains central."

Their work — supported by NIH grants and advanced computing resources — shows that the path forward isn't about rejecting AI, but about building it with humility, skepticism, and safeguards. Or, to put it another way: if we want AI in the clinic, we'd better teach it not to run with scissors.