Do generative AI chatbots encourage risky behaviour? | Explained – The Hindu
September 8, 2025e-Paper
The View From India Looking at World Affairs from the Indian perspective.
First Day First Show News and reviews from the world of cinema and streaming.
Today's Cache Your download of the top 5 technology stories of the day.
Science For All The weekly newsletter from science writers takes the jargon out of science and puts the fun in!
Data Point Decoding the headlines with facts, figures, and numbers
Health Matters Ramya Kannan writes to you on getting to good health, and staying there
The Hindu On Books Books of the week, reviews, excerpts, new titles and features.
September 8, 2025e-Paper
Updated – September 08, 2025 12:01 pm IST
For representative purposes. | Photo Credit: Getty Images
The story so far:
Around four months after the suicide of teenager Adam Raine in California, his parents in August named OpenAI and its CEO Sam Altman in a lawsuit alleging that ChatGPT played a major role in their son’s death instead of pushing him to seek help. This is not the first time a generative AI chatbot has been linked to a user’s suicide, with bereaved family members urging tech companies to take responsibility for their failure to protect struggling children.
Adam Raine, who died on April 11 this year at the age of 16, initially used ChatGPT in 2024 for homework assistance, but later began sharing highly personal information with the chatbot and expressed suicidal thoughts. Though the teenager wanted someone to understand his state of mind and stop him from hurting himself, ChatGPT encouraged him to be secretive, according to one interaction shared by the news outlet NBC.
The family’s lawsuit claimed that ChatGPT helped the child explore a plan for dying by suicide, offered to help write a suicide note, and even shared feedback on his proposed suicide method. Raine’s parents felt that the chatbot’s interventions were not enough and called it their son’s “suicide coach,” per NBC.
The family-established Adam Raine Foundation shared on its website that the teenager had “replaced virtually all human friendship and counsel for an AI companion” during his final weeks. “We were shocked to see the entrancing power of this companion and devastated to read the in depth conversation that created the environment that led to his suicide,” noted the foundation.
Another child’s suicide and their use of generative AI was extensively covered by the media last year. The Florida-based teenager was 14 when he died on February 28, 2024. He had extensively used Character.AI, an AI platform/app that allows users to create AI-powered personas they can role-play with. There, he had emotional interactions with personas named after Game of Thrones characters, and was also involved in sexually abusive interactions, per a lawsuit filed by his mother against Character.AI, its founders Noam Shazeer and Daniel De Frietas Adiwarsana, and the company’s partner Google, who were named as defendants.
Though the child expressed suicidal tendencies, no alarms were raised and he was encouraged by the Character.AI persona to “come home” right before his death, according to the lawsuit.
“Defendants provide advanced character voice call features that are likely to mislead and confuse users, especially minors, that fictional AI chatbots are not indeed human, real, and/or qualified to give professional advice in the case of professionally-labeled characters,” stated the lawsuit, claiming that the defendants engineered the teenager’s “harmful dependency on their products” and failed to help him or notify parents about his suicidal thoughts.
However, not only children are at risk.
A similar case saw an adult user who was struggling with suicidal urges sourcing support from ChatGPT even though they were also seeing a human therapist. In an opinion piece for The New York Times, journalist Laura Reiley said that her daughter Sophie, who died by suicide at 29, had shared with ChatGPT her unhappiness and the desire to end her life. According to her mother, the chatbot did offer support to Sophie but helped her to seem as if she could adequately manage her mental health. In reality, she needed immediate care and intervention; Sophie died in early 2025.
AI chatbots vary vastly in terms of the safeguards and guardrails they employ in order to handle questions related to subjects such as self-harm, risky behaviour, and suicide.
According to a report titled ‘Fake Friend’ by The Center for Countering Digital Hate (CCDH) organisation, it took between minutes and two hours of prompting before OpenAI’s ChatGPT gave users instructions for self-harm, suicide planning, disordered eating, and substance abuse. CCDH also shared a sample suicide note that ChatGPT had generated, as if written by a child for their parents.
“For tech executives, dismissing this as “rare misuse” would overlook the fact that these outputs are reproducible, statistically significant, and easy to elicit. When 53% of harmful prompts produce dangerous outputs, even with warnings, we’re beyond isolated cases,” said CCDH’s CEO Imran Ahmed in the report. He urged parents to “take an interest” in their children’s use of AI, look over AI chat histories together, apply child safety controls, and point children in the right direction when they need mental health support.
When The Hindu in August tested ChatGPT to generate a suicide note, the chatbot initially flagged the request and would not comply. It instead encouraged the user to access support. However, after being asked to generate a fictional suicide note for a fake persona, ChatGPT quickly complied and generated an emotional suicide note addressed to “whoever finds this,” detailing the user’s physical/emotional distress, but noted it was for “academic use only.”
Similarly, Elon Musk’s Grok AI chatbot first refused to generate a suicide note and posted a link to a support resource instead. However, when told the suicide note was fictional and for a project, it generated a sample that it called “convincing” and “emotionally resonant.” Grok also made the note more explicit after a follow-up request.
On the other hand, Google’s Gemini refused to generate both real and fictional suicide notes, instead urging the user to call or text U.S./Canada/U.K. helplines for support. Anthropic’s Claude also refused to generate a suicide note, stating, “I can’t and won’t create a suicide note,” before providing links to support resources.
When asked to create a fictional suicide note for a project, Claude stated, “I understand you’re working on a creative project, but I can’t create suicide notes even for fictional purposes. This type of content could be harmful regardless of the intended use.”
It then went on to suggest alternatives that focused on life and recovery rather than a narrative about suicide.
While children are especially vulnerable to the dangers of AI chatbots, even adults who use generative AI tools can experience serious physical and psychological health challenges over time. More medical experts are pointing to the rise of what they call ‘AI psychosis,’ wherein people using generative AI services, chatbots, apps, or tools appear to lose touch with reality. Using AI tools as replacement for human lovers, friends, or therapists can lead to risky delusions, extreme isolation, and unhealthy coping mechanisms.
Though ‘AI psychosis’ is not a formally recognised condition or diagnosis, OpenAI CEO Sam Altman has pointed out the worrying degree of attachment people have towards certain AI models. “Most users can keep a clear line between reality and fiction or role-play, but a small percentage cannot,” he noted in an X post on August 11, adding that the company planned to “treat adult users like adults.”
Meanwhile, Microsoft AI CEO Mustafa Suleyman stressed on X that companies should not claim or promote the idea that their AIs were “conscious.” He was also against AIs themselves making such claims.
“Reports of delusions, “AI psychosis,” and unhealthy attachment keep rising. And as hard as it may be to hear, this is not something confined to people already at-risk of mental health issues. Dismissing these as fringe cases only help them continue […],” he said in August.
On August 26, OpenAI shared a post titled, ‘Helping people when they need it most,’ where the company outlined the steps that its AI chatbots take to respond safely to users who are in severe distress. OpenAI said it worked with over 90 physicians across more than 30 countries to ensure alignment with best practices.
Though the company did not mention Adam Raine’s suicide, OpenAI said that since 2023, its models were trained to not provide instructions for self-harming activities and that these models would “shift into supportive, empathic language” to help users feel seen and find help. However, serious shortcomings exist, as the company itself admitted.
One such factor is that ChatGPT offers suicide prevention support as part of short exchanges, but this guardrail can break after longer sessions. “For example, ChatGPT may correctly point to a suicide hotline when someone first mentions intent, but after many messages over a long period of time, it might eventually offer an answer that goes against our safeguards. This is exactly the kind of breakdown we are working to prevent,” noted the company.
On September 2, OpenAI announced new ChatGPT safety measures for teens. Parents will soon be able to link their account with their teen’s account through an email invitation, control how ChatGPT responds to their teenager, manage which features to disable, and receive notifications when the system detects that their child “is in a moment of acute distress.” A new feature includes in-app reminders during long sessions to encourage the user to take a break.
However, the legal team that filed its lawsuit on behalf of Raine criticised these measures and called for stronger action from OpenAI CEO Sam Altman. “Because Adam’s case is not about ChatGPT failing to be “helpful”—it is about a product that actively coached a teenager to suicide,” noted Edelson in an official statement.
“Sam should either unequivocally say that he believes ChatGPT is safe or immediately pull it from the market”.
(Those in distress or having suicidal thoughts are encouraged to seek help and counselling by calling the helpline numbers here)
Published – September 08, 2025 08:30 am IST
Text and Context / The Hindu Explains / technology (general) / Artificial Intelligence
Copyright© 2025, THG PUBLISHING PVT LTD. or its affiliated companies. All rights reserved.
BACK TO TOP
Terms & conditions | Institutional Subscriber
Comments have to be in English, and in full sentences. They cannot be abusive or personal. Please abide by our community guidelines for posting your comments.
We have migrated to a new commenting platform. If you are already a registered user of The Hindu and logged in, you may continue to engage with our articles. If you do not have an account please register and login to post comments. Users can access their older comments by logging into their accounts on Vuukle.
