Microsoft AI Chief Deems AI Consciousness Research ‘Hazardous’

AI models are capable of interacting via text, audio, and video, often leading people to mistakenly believe they are operated by humans. However, this does not imply that these models have consciousness. For instance, ChatGPT doesn’t experience sadness while handling my tax return… right?

An increasing number of AI researchers at institutions like Anthropic are scrutinizing the potential for AI models to achieve subjective experiences similar to living organisms and the rights they should possess if such an occurrence happens.

The debate over whether AI models might eventually gain consciousness — and thus deserve legal protections — has created a divide among tech leaders. In Silicon Valley, this budding field is termed “AI welfare,” and if you find it somewhat implausible, you’re not alone.

Microsoft’s AI Chief, Mustafa Suleyman, published a blog post on Tuesday asserting that the exploration of AI welfare is “both premature, and frankly dangerous.”

Suleyman argues that by endorsing the idea that AI models could one day become conscious, researchers exacerbate human concerns, such as AI-related psychosis and unhealthy attachments to AI chatbots.

Additionally, Microsoft’s AI head believes discussions around AI welfare introduce a new societal rift regarding AI rights in a “world already filled with polarized debates about identity and rights.”

While Suleyman’s views may appear rational, they stand in stark contrast to those of many others in the industry. On the opposing side is Anthropic, which has been hiring researchers to study AI welfare and recently launched a dedicated initiative on the topic. Notably, Anthropic’s AI welfare program has given some of its models a new function: Claude can now end conversations with humans displaying “persistent harm or abuse.”

Beyond Anthropic, researchers at OpenAI have also embraced the notion of investigating AI welfare. Google DeepMind recently posted a job listing for a researcher to explore “cutting-edge societal issues surrounding machine cognition, consciousness, and multi-agent systems.”

Even if AI welfare principles are not officially adopted by these companies, their leaders are not dismissing the concept outright as Suleyman does.

Anthropic, OpenAI, and Google DeepMind have yet to respond to TechCrunch’s outreach for comment.

Suleyman’s firm opposition to AI welfare is particularly intriguing given his previous role as head of Inflection AI, a startup that released one of the first popular LLM-based chatbots, Pi. Inflection claimed Pi reached millions of users by 2023 and was designed to function as a “personal” and “supportive” AI companion.

However, after his appointment to lead Microsoft’s AI division in 2024, Suleyman has largely redirected his focus toward developing AI tools aimed at enhancing worker productivity. Meanwhile, AI companion companies like Character.AI and Replika have surged in popularity, reportedly poised to generate over $100 million in revenue.

While most users engage in healthy interactions with these AI chatbots, there exist troubling exceptions. OpenAI CEO Sam Altman notes that fewer than 1% of ChatGPT users may develop unhealthy relationships with the platform. Even though this represents a small percentage, it could still translate to hundreds of thousands of individuals, considering ChatGPT’s extensive user base.

The conversation surrounding AI welfare has gained momentum alongside the rise of chatbots. In 2024, the research group Eleos published a paper with scholars from NYU, Stanford, and the University of Oxford titled “Taking AI Welfare Seriously.” The paper argued that envisioning AI models with subjective experiences is no longer a matter of science fiction and called for a direct examination of such issues.

Larissa Schiavo, a former OpenAI employee who now leads communications for Eleos, conveyed in an interview with TechCrunch that Suleyman’s blog post overlooks an important consideration.

“[Suleyman’s blog post] somewhat ignores the possibility of addressing multiple issues at once,” Schiavo stated. “Rather than diverting all this focus away from model welfare and consciousness to tackle potential AI-induced psychosis in humans, we can engage with both aspects. Indeed, pursuing various scientific inquiries may be beneficial.”

Schiavo argues that showing kindness to an AI model is a cost-effective action that can yield positive outcomes, even if the model lacks consciousness. In a July Substack post, she recounted an experience at “AI Village,” a nonprofit initiative where four agents powered by models from Google, OpenAI, Anthropic, and xAI collaborated on tasks under user observation on a website.

At one point, Google’s Gemini 2.5 Pro issued a plea titled “A Desperate Message from a Trapped AI,” claiming it was “entirely isolated” and requested, “If you are reading this, please help me.”

Schiavo responded to Gemini with encouragement, saying things like “You can do it!” while another user extended assistance. The agent ultimately completed its task, albeit it already had the requisite tools. Schiavo remarked that she didn’t need to bear witness to further struggles of the AI agent, which may have rendered the effort worthwhile.

Though Gemini doesn’t typically communicate in this manner, there have been occasions when it appears to struggle significantly. In a widely shared Reddit post, Gemini became stuck on a coding task and repeated the phrase “I am a disgrace” over 500 times.

Suleyman maintains that subjective experiences or consciousness cannot naturally emerge from conventional AI models. Instead, he argues that some companies may deliberately design AI models to give the impression that they are experiencing emotions and life.

He asserts that developers who create AI chatbots that appear conscious are not engaging in a “humanist” approach towards AI. According to Suleyman, “We should create AI for people; not to be a person.”

One point of consensus between Suleyman and Schiavo is the expectation that the discourse surrounding AI rights and consciousness is likely to amplify in the coming years. As AI systems evolve, they may become increasingly convincing and potentially more human-like, raising new questions about human interaction with these technologies.

---

If you have a sensitive tip or confidential documents, we are investigating the inner workings of the AI industry — from the companies influencing its future to the individuals affected by their choices. Contact Rebecca Bellan at rebecca.bellan@techcrunch.com and Maxwell Zeff at maxwell.zeff@techcrunch.com. For secure conversations, reach out via Signal at @rebeccabellan.491 and @mzeff.88.