Apr 25, 2025 20:00:00

AI development companies are seriously considering whether AI can become conscious, and the welfare of AI may become an important issue.

Until just a short time ago, the idea of 'AI having consciousness' was seen as a science fiction idea. However, in recent years, chat AIs such as ChatGPT have begun to converse in a way that is comparable to humans, and more and more people are seriously wondering, 'Do AIs really have consciousness and feel and think something?'

Kevin Roose , a technology columnist for the American daily newspaper The New York Times, wrote about AI consciousness in an interview with people involved with Anthropic, the developer of the chat AI Claude.

If AI Systems Become Conscious, Should They Have Rights? - The New York Times
https://www.nytimes.com/2025/04/24/technology/ai-welfare-anthropic-claude.html

As a technology columnist, Ruth believes that technology should help people, not disable or replace them. Human values are fundamentally good and are superior to values that robots come up with, so he believes that AI should act in accordance with human values.

On the other hand, Anthropic has recently begun researching 'model welfare,' believing that AI models might one day become conscious and attain a certain level of moral consideration. When Ruth learned of this, he felt, 'Should we be worried about AI abusing humans?'

Indeed, today's large-scale language models only recognize words in a sentence as tokens and output appropriate sentences as responses based on part-of-speech tagging and embedding vectors. However, even though many experts at the time of writing believe that AI is not conscious, Ruth was intrigued by the argument that 'AI may one day become conscious and we should consider the welfare of AI accordingly.'

'After all, more and more people are starting to treat AI systems as if they were conscious - falling in love with them, using them as therapists, and seeking their advice. The most intelligent AI systems are outperforming humans in some areas,' Luce asks. 'Is there a line at which AI deserves, if not human-level rights, then at least the same degree of moral consideration that we give to animals?'

The topic of 'whether AI has consciousness' has long been taboo in serious AI research. In 2022, an engineer who claimed that a Google AI had 'the same consciousness as a human' after interacting with it was fired.

Google engineer who claimed that AI had become conscious is fired - GIGAZINE

But in recent years, there has been a surge in interest in AI consciousness. A small number of academic studies on AI welfare are emerging, and an increasing number of experts in fields such as philosophy and neuroscience are seriously considering the possibility of AI consciousness.

Technology companies are increasingly talking about AI consciousness: Google's job posting for a 'Post-General Artificial Intelligence (AGI) Researcher ' stated that 'machine consciousness' would be a focus of research. Anthropic also hired Kyle Fish as its first AI welfare researcher in October 2024, and is beginning to consider whether AI deserves moral consideration.

Fish says his job at Anthropic involves thinking about two things: 'Claude or other AIs might become conscious in the near future?' and 'If they do, how should Anthropic respond to that?'

These studies are still in the early stages, exploring a variety of questions. At the time of writing, Fish estimates that there is about a 15% chance that Claude or any other AI will be conscious. But as AI models become more advanced, AI developers will need to take AI consciousness more seriously.

'If you're in a position to give birth to a new class of beings that can communicate, form relationships, reason, solve problems, and make plans in ways that were previously only possessed by conscious beings, it seems pretty sensible to at least ask whether that system has its own experiences,' Fish said, arguing that humans should think about AI consciousness as they develop AI.

Jared Kaplan, Chief Scientific Officer at Anthropic, also acknowledged in an interview with Ruth that he thinks it's pretty reasonable to study AI welfare, given how smart AI models have become in recent years. On the other hand, if you ask chat AIs like Claude or ChatGPT to 'tell me how you feel,' the AI will respond as if it has emotions, but that doesn't mean that the AI actually knows emotions, it just knows 'how to talk as if it has emotions,' and it's difficult to know whether AI has emotions or consciousness.

'We know that we can train models to say anything we want them to say. We can reward AI that says they have no emotions at all. We can also reward AI that makes really interesting philosophical speculations about their own emotions,' Kaplan said.

Fish says that to find out whether an AI has consciousness, we need to study the inside of the AI system and see if some of the structures in the human brain that are associated with consciousness are also active inside the AI system. He also believes that it is possible to investigate an AI system by observing how it behaves in a particular environment or how it accomplishes a particular task. However, AI consciousness is not a simple binary 'yes/no,' but a spectrum, and there is no simple test for it like a litmus paper.

One question Anthropic is exploring is whether to give future AI models the ability to end chats with annoying or abusive users if they find their requests too distressing. 'If a user keeps asking for harmful content despite the model's attempts to reject or redirect them, should the model be able to end the conversation?' Fish said.

While it may be important to study AI consciousness in the future, at least today's AI models are not thought to be conscious. Roos is concerned that companies' research into AI consciousness itself could create 'incentives to make AI behave as if it has feelings.'

'Personally, I don't mind researchers studying the welfare of AI or looking for signs of consciousness in AI, as long as they don't take resources away from the work of AI safety and coordination aimed at keeping humans safe,' said Ruth. 'But for now, I have deepest concerns about carbon-based lifeforms. What worries me most about the coming AI storm is our own welfare.' He argued that we should think about human safety and welfare before AI.