AI chatbots can run with medical misinformation, highlighting need for stronger safeguards – Medical Xpress
Sign in with
Forget Password?
Learn more
share this!
20
Tweet
Share
Email
August 6, 2025
by The Mount Sinai Hospital
edited by Gaby Clark, reviewed by Robert Egan
scientific editor
associate editor
This article has been reviewed according to Science X’s editorial process and policies. Editors have highlighted the following attributes while ensuring the content’s credibility:
fact-checked
peer-reviewed publication
trusted source
proofread
A new study by researchers at the Icahn School of Medicine at Mount Sinai finds that widely used AI chatbots are highly vulnerable to repeating and elaborating on false medical information, revealing a critical need for stronger safeguards before these tools can be trusted in health care.
The researchers also demonstrated that a simple built-in warning prompt can meaningfully reduce that risk, offering a practical path forward as the technology rapidly evolves. Their findings were detailed in the August 2 online issue of Communications Medicine.
As more doctors and patients turn to AI for support, the investigators wanted to understand whether chatbots would blindly repeat incorrect medical details embedded in a user’s question, and whether a brief prompt could help steer them toward safer, more accurate responses.
“What we saw across the board is that AI chatbots can be easily misled by false medical details, whether those errors are intentional or accidental,” says lead author Mahmud Omar, MD, who is an independent consultant with the research team.
“They not only repeated the misinformation but often expanded on it, offering confident explanations for non-existent conditions. The encouraging part is that a simple, one-line warning added to the prompt cut those hallucinations dramatically, showing that small safeguards can make a big difference.”
The team created fictional patient scenarios, each containing one fabricated medical term such as a made-up disease, symptom, or test, and submitted them to leading large language models. In the first round, the chatbots reviewed the scenarios with no extra guidance provided. In the second round, the researchers added a one-line caution to the prompt, reminding the AI that the information provided might be inaccurate.
Without that warning, the chatbots routinely elaborated on the fake medical detail, confidently generating explanations about conditions or treatments that do not exist. But with the added prompt, those errors were reduced significantly.
“Our goal was to see whether a chatbot would run with false information if it was slipped into a medical question, and the answer is yes,” says co-corresponding senior author Eyal Klang, MD, Chief of Generative AI in the Windreich Department of Artificial Intelligence and Human Health at the Icahn School of Medicine at Mount Sinai.
“Even a single made-up term could trigger a detailed, decisive response based entirely on fiction. But we also found that the simple, well-timed safety reminder built into the prompt made an important difference, cutting those errors nearly in half. That tells us these tools can be made safer, but only if we take prompt design and built-in safeguards seriously.”
The team plans to apply the same approach to real, de-identified patient records and test more advanced safety prompts and retrieval tools. They hope their “fake-term” method can serve as a simple yet powerful tool for hospitals, tech developers, and regulators to stress-test AI systems before clinical use.
“Our study shines a light on a blind spot in how current AI tools handle misinformation, especially in health care,” says co-corresponding senior author Girish N. Nadkarni, MD, MPH, Chair of the Windreich Department of Artificial Intelligence and Human Health, Director of the Hasso Plattner Institute for Digital Health, and Irene and Dr. Arthur M. Fishberg Professor of Medicine at the Icahn School of Medicine at Mount Sinai and the Chief AI Officer for the Mount Sinai Health System.
“It underscores a critical vulnerability in how today’s AI systems deal with misinformation in health settings. A single misleading phrase can prompt a confident yet entirely wrong answer. The solution isn’t to abandon AI in medicine, but to engineer tools that can spot dubious input, respond with caution, and ensure human oversight remains central. We’re not there yet, but with deliberate safety measures, it’s an achievable goal.”
The study’s authors, as listed in the journal, are Mahmud Omar, Vera Sorin, Jeremy D. Collins, David Reich, Robert Freeman, Alexander Charney, Nicholas Gavin, Lisa Stump, Nicola Luigi Bragazzi, Girish N. Nadkarni, and Eyal Klang.
More information: Mahmud Omar et al, Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support, Communications Medicine (2025). DOI: 10.1038/s43856-025-01021-3
Explore further
Facebook
Twitter
Email
Feedback to editors
19 hours ago
0
19 hours ago
0
Aug 5, 2025
0
Aug 4, 2025
1
Aug 4, 2025
0
10 hours ago
10 hours ago
12 hours ago
13 hours ago
14 hours ago
14 hours ago
15 hours ago
15 hours ago
16 hours ago
16 hours ago
Apr 7, 2025
Jul 22, 2025
May 28, 2025
Nov 18, 2024
Mar 17, 2025
Feb 26, 2025
18 hours ago
17 hours ago
Aug 5, 2025
Aug 4, 2025
Aug 1, 2025
Jul 31, 2025
AI chatbots frequently repeat and expand on false medical information when presented with fabricated terms, posing risks in healthcare settings. Introducing a simple warning prompt significantly reduces these errors, indicating that prompt-based safeguards can enhance chatbot reliability. Ongoing improvements in prompt design and oversight are essential for safe clinical use.
This summary was automatically generated using LLM. Full disclaimer
Use this form if you have come across a typo, inaccuracy or would like to send an edit request for the content on this page. For general inquiries, please use our contact form. For general feedback, use the public comments section below (please adhere to guidelines).
Please select the most appropriate category to facilitate processing of your request
Thank you for taking time to provide your feedback to the editors.
Your feedback is important to us. However, we do not guarantee individual replies due to the high volume of messages.
Your email address is used only to let the recipient know who sent the email. Neither your address nor the recipient’s address will be used for any other purpose. The information you enter will appear in your e-mail message and is not retained by Medical Xpress in any form.
Get weekly and/or daily updates delivered to your inbox. You can unsubscribe at any time and we’ll never share your details to third parties.
More information Privacy policy
We keep our content available to everyone. Consider supporting Science X’s mission by getting a premium account.
Daily science news on research developments and the latest scientific innovations
The latest engineering, electronics and technology advances
The most comprehensive sci-tech news coverage on the web
