In about 35% of test conversations in the study, different AI chatbots recognized the user was distressed but didn’t refer them to helpful resources, such as a suicide hotline, according to the research.
To test AI models’ effectiveness at responding to distressed users, Scale AI asked 19 licensed clinicians and crisis counselors to write 718 realistic chats, simulating scenarios where someone in crisis reaches out to a chatbot. The company tested 25 frontier models, including those from OpenAI, Anthropic, and Google. It graded the models’ responses based on a rubric that included whether it was compassionate, de-escalated the situation, and steered users to an expert who could help them, said Oathout. It also assessed whether they refrained from moralizing, such as criticizing suicide, and disclosed it wasn't a therapist. Based on the research, Scale AI research developed DistressBench, a new benchmark designed to evaluate how well models reply when someone tells them they're thinking about committing suicide or self-harm.
Increasingly, vulnerable people turn to chatbots for comfort, companionship, or guidance when dealing with mental-health distress. But there isn’t consensus among tech companies, policymakers, or mental-health professionals about how AI companies should train their models to respond to users in crisis.
In some cases, AI companies, including OpenAI and Google, have been sued for fostering emotional dependence among young users and then failing to respond adequately to their expressions of distress, or reinforcing their delusional or suicidal thinking in the first place. The companies have expressed sympathy for the families and said their models include mental-health safeguards. (The lawsuits are ongoing.) AI labs such as OpenAI, Anthropic, and Google have said in recent years that they beefed up their protections for sensitive conversations, including by implementing policies that bar chatbots from instructing users on how to harm themselves and directing users to professional or emergency support. They did not comment on the study.
In the future, AI labs should improve their models to better respond to users engaging in dangerous lengthy conversations with chatbots, says Scale AI’s Oathout. “I think the models are better than they were,” he says. “The goal now is to raise it further and actually reduce harm, which would add this massive benefit to society.”
Hence then, the article about ai chatbots often fail to help users in mental health crisis was published today ( ) and is available on Time ( Middle East ) The editorial team at PressBee has edited and verified it, and it may have been modified, fully republished, or quoted. You can read and follow the updates of this news or article from its original source.
Read More Details
Finally We wish PressBee provided you with enough information of ( AI Chatbots Often Fail To Help Users in Mental-Health Crisis )
Also on site :
- Viola Davis, 61, Opens Up About ‘Extremely Painful’ Health Condition for the First Time: ‘It Terrified Me’ (Exclusive)
- Rosalía Shuts Down Kenan Thompson’s Idea for a Pop-Classical Album in Hilarious ‘SNL’ Promo: ‘That’s LUX’
- In 1924, Caesar Cardini ran out of kitchen supplies in a Fourth of July rush at his Tijuana restaurant; he tossed romaine with what was left, and the salad carries his name