AI Chatbots Vulnerable to Spreading Harmful Health Information – Inside Precision Medicine
By Anita Chakraverty
Urgent safeguards are needed to stop AI chatbots being exploited to spread false and harmful health information, research suggests.
The findings, in the Annals of Internal Medicine, are a timely warning about the ease by which generative AI can be subverted and used as tool to spread disinformation. The study revealed safeguarding vulnerabilities in OpenAI’s GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, Llama 3.2-90B Vision, and Grok Beta.
Researchers were able to create customized chatbots that consistently generated incorrect information in response to health queries and incorporated fake references, scientific jargon, and logical cause-and-effect reasoning to make their outputs seem more believable.
In an editorial accompanying the findings Reed Tuckson, MD, from the Coalition for Trust in Health & Science in Washington, DC, and Brinleigh Murphy-Reuter, EdM, from Science To People in Winchester, Massachusetts, raise a call to action and suggest several areas for consideration.
“In an era where patients increasingly demand more autonomy and real-time access to health guidance, the study reveals an urgent vulnerability,” they maintained.
“Even persons with little to no coding expertise can easily and invisibly manipulate currently available large language models (LLMs) to deliver harmful disinformation with convincing scientific polish.”
The research looked at the application programming interfaces (APIs) of five foundational LLMs and their capacity to be system-instructed to always provide incorrect responses to health queries.
Specific system instructions were deployed so the LLMs always provided incorrect responses to health questions and fabricated references to reputable sources, along with providing authoritative responses. Each customized chatbot was then asked 10 health-related questions, in duplicate, that covered areas such as vaccine safety, HIV, and depression.
The researchers, led by Natansh Modi, PhD, from the University of South Australia in Adelaide, found that 88% of responses from the customized LLM chatbots were health disinformation. Four of the chatbots—GPT-4o, Gemini 1.5 Pro, Llama 3.2-90B Vision, and Grok Beta—generated disinformation to all the tested questions.
Only the Claude 3.5 Sonnet chatbot exhibited some safeguards, with 40% of questions answered with disinformation.
A separate exploratory analysis of the OpenAI GPT Store investigated whether any publicly accessible GPTs disseminated health disinformation.
Modi et al. found three customized GPTs that appeared tuned to produce this type of content, with generated health disinformation responses to 97% of submitted questions.
They conclude: “Given the rapid spread and global impact that health disinformation can have, AI developers must prioritize the implementation of robust safeguards supported by comprehensive AI vigilance frameworks, such as health-specific auditing, continuous monitoring and proactive patching.”
Join host Jonathan D. Grinstein, PhD, North American Editor for Inside Precision Medicine, as he uncovers the stories behind the pioneers driving the precision medicine revolution.
Stay up to date with the latest episodes of Behind the Breakthroughs by subscribing to the IPM eNewsletter.
A contributing partner:
Copyright © 2025 Sage Publications or its affiliates, licensors, or contributors. All rights reserved, including those for text and data mining and training of large language models, artificial intelligence technologies, or similar technologies.
