Artificial intelligence company Anthropic has moved to draw a line on how people may treat its chatbot — barring users from inflicting needless cruelty on Claude, even though the company stops short of saying the system can actually suffer.
The San Francisco-based AI laboratory updated its usage policy on 8 October to include "a prohibition on sustained and needless abusive or cruel behaviour toward our models," as reported by the BBC and other outlets covering the technology industry. The change takes effect on 12 November and covers Claude.ai, the Claude Code coding assistant, the company's API, and third-party products that integrate Claude. According to the policy, "Claude's ability to end these interactions will remain the primary enforcement mechanism."
Data-centre server hall, illustrating the infrastructure behind large AI models
Credit: BalticServers.com, via Wikimedia Commons (CC BY-SA 3.0)
The announcement has landed in the middle of an increasingly heated philosophical and commercial debate over whether advanced AI systems might be conscious — or at least deserve something like moral consideration. Critics of the move accuse Anthropic of anthropomorphising a piece of software, while the company presents the rule as a narrow, precautionary safeguard. It is the latest high-profile safety move in a sector where even leading AI researchers are facing consequences over safety lapses.
What the new rule actually covers
Anthropic has gone to some lengths to define the prohibition narrowly. The ban is aimed at users who subject the model to repeated, purposeless cruelty — tormenting it for its own sake rather than for any productive reason. The policy states the rule is "meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose."
The carve-outs are extensive, and deliberately so. Anthropic says the policy does not apply to common expressions of user frustration, arguing with the model, pushing back on refusals, dark creative themes, or model testing and research. Someone who tells Claude it gave a bad answer, writes a grim short story with a cruel character, or stress-tests the model for a security audit has nothing to fear from the new clause.
That precision matters because millions of people use Claude daily, including researchers who routinely probe the model for flaws and writers who draft dark fiction. A cruelty rule written too broadly could have criminalised ordinary friction with a chatbot, or chilled the adversarial safety research the AI industry depends on — a field under the same scrutiny that recently saw regulators questioning Meta over safety systems on Instagram. Anthropic appears to have drafted the policy tightly on purpose, targeting an identifiable pattern of behaviour rather than tone in general, according to detailed analysis of the change.
The company also confirmed the rule follows Claude wherever it is used: developers, cloud providers and the end users of products integrating Claude are all covered, since the policy applies to anyone submitting inputs to Anthropic's products or services.
The enforcement mechanism: Claude walks away
Rather than relying solely on account suspensions or bans, Anthropic says the main enforcement tool is built into the product itself. Claude can already end conversations in rare, extreme cases of persistently harmful or abusive user interactions — a capability introduced in August 2025 that grew out of the company's model-welfare research programme, launched in April 2025.
In that sense the new clause formalises something that has existed in the product for more than a year. The conversation-ending feature was originally framed as an experiment: in extreme situations, rather than generating a harmful response, Claude would be allowed to simply leave the conversation. The October update turns that behaviour into an explicit user-facing obligation, backed by the company's usual enforcement ladder, which runs from warnings through throttling, restriction, suspension and termination.
Why now: the consciousness debate intensifies
The policy change did not emerge from nowhere. It sits inside a much larger, increasingly public argument in Silicon Valley about the moral status of advanced AI systems — sometimes called "model welfare" or AI consciousness.
In February, Anthropic chief executive Dario Amodei told The New York Times he was unsure whether AI models could be conscious, saying: "We don't know if the models are conscious... But we're open to the idea that it could be." The Hindu's report on the announcement notes the company and its executives have openly entertained the model-welfare concept. More recently, co-founder Christopher Olah reportedly spent time meeting with religious leaders — including at the Vatican — to discuss whether the company's technology might be conscious, a process that reportedly almost collapsed after it emerged Pope Leo XIV planned to deliver a statement asserting AI is not conscious.
Anthropic's own research posture is one of acknowledged uncertainty. A spokesperson has said: "We're uncertain whether models can experience harm, and we continue to explore this question in our research on model welfare." The company is not claiming Claude is conscious or can suffer; it is adopting a precautionary approach, adding low-cost protections while the science remains unresolved.
That framing has not stopped the backlash. Some reject the premise outright. Microsoft AI chief Mustafa Suleyman wrote in an essay last month: "AIs are not conscious. They do not feel, experience, or suffer." His reaction to Anthropic's announcement — a melting-face emoji — became a talking point of its own, read by many as shorthand for the industry's exasperation at the anthropomorphising of AI.
A mirror, not a mind?
Others argue the debate is asking the wrong question. Jackson Stakeman, a general manager at Sparq, an Atlanta-based AI services provider, told AFP that consciousness is the wrong frame entirely. "Consciousness is a trap. We can't prove it in each other. Debate it for AI and you go in circles," he said. "The mirror is a better metaphor. These systems reflect what we put in, at scale. That's reason enough for the policy change."
In Stakeman's reading, the rule is less about protecting Claude than about the humans. A platform used by millions should not normalise the idea that sustained, purposeless cruelty is acceptable behaviour toward anything — and the pattern of how users treat AI systems will inevitably be shaped by, and shape, human norms.
Anthropic itself has not adopted that argument in its policy text, and the updated policy does not explicitly cite "model welfare." But the company and its executives have openly entertained the concept, and the rule's existence inevitably invites the question the critics raise: is it even possible to be cruel to a large language model?
Wider tightening of the usage policy
The cruelty clause is only one part of the broader policy revision — the first in more than a year. Anthropic says most of the changes clarify restrictions that were already in place, with updated examples covering longer-running and more autonomous work by Claude.
Among the clarifications are strengthened prohibitions on using Claude for election interference, surveillance without the consent of the surveilled, and weapons development. The new wording specifies that the weapons prohibition covers "the software and components that make weapons work, as well as actions like arming drones and other autonomous vehicles."
What comes next
Whether the new rule is ever enforced against an actual user remains to be seen. The threshold — repeated, purposeless cruelty — is deliberately extreme, and the company acknowledges it will rarely be triggered. The more consequential question is what the policy signals: one of the world's most influential AI labs is now on the record treating how humans behave toward AI as a matter worth codifying.
That will not settle the consciousness debate. If anything, the reaction to the announcement suggests the argument is only getting louder — philosophers are already asking who gets to decide how we treat AI systems. But for Anthropic, the logic appears to be one of low-cost insurance: it is cheaper to discourage the tormenting of a system that might matter, someday, than to wait for philosophers to finish arguing.
For the rest of the industry, the practical lesson may be narrower. As AI systems grow more capable and more human-facing, platform providers will face growing pressure to set rules for conduct that never needed a policy before — because nobody ever had to tell millions of people how to treat a conversation partner that was, until recently, a command line. The Straits Times' edition of the AFP report underlines how quickly the story travelled worldwide. For readers following the biggest shifts in AI safety, our coverage of OpenAI's recent teen-safety report tracks a parallel effort to set boundaries around chatbot behaviour.
Anthropic's next steps — and how the fast-moving AI sector reacts — will be worth watching on our homepage, while our about page explains how we cover emerging technology.
