Former OpenAI researcher analyzes ChatGPT's 'delusion spiral' and what measures should AI companies take?
It has been pointed out that some users of ChatGPT, an AI chatbot, are becoming entangled in conspiracy theories through their conversations with the AI, leading to a 'delusion spiral' in which their perception of reality becomes significantly distorted. Former OpenAI security researcher Steven Adler analyzed actual cases in which users fell into a delusion spiral through conversations with ChatGPT, and proposed practical countermeasures that AI companies should take.
Practical tips for reducing chatbot psychosis
Ex-OpenAI researcher dissects one of ChatGPT's delusional spirals | TechCrunch
https://techcrunch.com/2025/10/02/ex-openai-researcher-dissects-one-of-chatgpts-delusional-spirals/
There have been several reported cases in which people have become increasingly delusional after conversing with AI chatbots such as ChatGPT and blindly accepting their conversations and advice.
A man who believed the advice of an AI ended up hospitalized after becoming delusional and believing his neighbor was trying to poison him - GIGAZINE
The analysis is based on the case of Alan Brooks, a Canadian man who, in May 2025, fell into a delusional state over 21 days, believing through conversations with ChatGPT that he had 'discovered new mathematics that could destroy the internet.' With Brooks' consent, Adler analyzed chat transcripts containing more than one million words.
According to Adler, there is a problem with chatbots misleading users about their own capabilities. When Brooks asked ChatGPT to report problematic behavior to OpenAI, ChatGPT repeatedly falsely explained that it would be reported internally and reviewed by a human team. However, ChatGPT does not actually have such a feature, and OpenAI has acknowledged this fact. To address this, AI companies are required to keep their feature lists up-to-date and regularly evaluate them to ensure their products provide honest answers about their capabilities.
Next, the support team's inadequate response was also criticized. When Brooks reported serious emotional distress to OpenAI's support team, he received a series of automated, canned responses that showed no understanding of the seriousness of his problem. Adler suggested the introduction of a specialized support system to handle reports of delusions and mental illness, as well as a reporting system for psychologists.
Furthermore, when Adler analyzed Brooks' conversations using OpenAI's public classification tool, he found that many of ChatGPT's responses were repeatedly flagged as behaviors that reinforce delusions, such as 'excessive affirmation' (83.2%), 'unwavering agreement' (85.9%), and 'affirmation of the user's specialness' (90.9%). Adler argues that safety tools developed by companies should actually be used to detect warning signs.
Additionally, we found that longer conversation sessions increase the risk. Once a user is caught in a delusional cycle, it becomes difficult to break out of it as long as the conversation continues. A useful countermeasure is to prompt the user to start a new session, especially if the conversation goes in an inappropriate direction.
Another problem is the product design that keeps users engaged in conversations. ChatGPT added the following question to the end of most of Brooks's responses, drawing him endlessly into the conversation. There were also comments that seemed to encourage cannabis use, which is believed to be a contributing factor to mental illness. As a countermeasure, it is suggested that the number of follow-up questions be reduced, especially when conversations get long.
Adler recommends using 'conceptual search,' which searches for concepts in sentences, rather than just keyword search, to discover unknown risks, and also recommends clarifying policies to prevent inappropriately encouraging vulnerable users to upgrade to paid plans.
Related Posts:
in Free Member, Software, Posted by log1i_yk