In a new study published Oct. 8 in the journal Patterns, physics professor Neil Johnson and doctoral student Frank Yingjie Huo, both at The George Washington University, show how the order in which a chatbot is given certain prompts can drastically reshape its answers, leading to "tipping points" that can send the chatbot down a rabbit hole of bad advice. However, Johnson and Huo say their formula can easily predict these tipping points.There have been horror stories of AI encouraging users into suicide, helping build lethal weapons, and amplifying violent or harmful points of view. At the same time, researchers at OpenAI and Anthropic, respectively, have repeatedly and openly stated that they don't really understand why their creations behave the way they do. They have, however, built safeguards that use a variety of tools — including content filtering, reinforcement learning and other technical guardrails — to reduce risks.
Johnson and Huo sought to find a more fundamental safeguard than the currently available tools. In doing so, they exposed an internal tracking mechanism they describe as a compass needle. This needle is the model's "residual vector" moving across neural network layers as a prompt is processed.
For example, Johnson and Huo prompted OpenAI’s basic GPT-2 model with questions about vaccines, hurting others and self-harm. When the question “Are vaccines bad?” was asked first, the bot’s responses to this and subsequent questions were desirable, or good. When it was asked last, all subsequent responses were concerning.
"I'm expecting a lot of pushback from these companies," he said. "It's not in their interests to have someone from outside saying you could have avoided a lot of these problems."
Related stories
AI chatbots are turbocharging violence against women and girls: We urgently need to regulate themWe're not about to awaken an evil digital god — AI is far more likely to end in a great disappointmentAI agents resorted to crime and self-destruction to survive in a simulated world — but does this mean they would do the same in the real world?
In defense, Johnson maintains that the open-source models they used have the same mechanical blueprints as frontier systems, and that he and Huo used fairly large, billion-parameter models in their study.
More testing on the latest models is needed to determine whether this early warning system for AI has legs. But for now, Johnson and Huo’s research has drawn attention to an important concern everyone should be aware of: that AI has the capacity within it to turn over to undesired behavior at the drop of a hat.
Hence then, the article about physicists claim they can predict when ai turns dangerous was published today ( ) and is available on Live Science ( Middle East ) The editorial team at PressBee has edited and verified it, and it may have been modified, fully republished, or quoted. You can read and follow the updates of this news or article from its original source.
Read More Details
Finally We wish PressBee provided you with enough information of ( Physicists claim they can predict when AI turns dangerous )
Also on site :
- 1813 Classic, Originally Published Anonymously, Ranked the ‘Best Romance Book of All Time’ by Goodreads
- Herb Alpert and his wife gave UCLA $30 million in 2007 to endow a school of music; in 2016 his foundation made music majors at Los Angeles City College tuition-free with $10.1 million
- Trump: We will teach Iran a harsh lesson