Confession Is Not a Bug Fix

Picture a chatbot in a confessional. It gave a customer the wrong refund policy, or the wrong medication schedule, or a court deadline that doesn't exist, and now it is profoundly sorry. It describes its failure in moving detail. It promises to do better. Someone on the other side of the screen weighs the sincerity of its repentance. The customer would probably settle for a correction.

I didn't make this up for the newsletter. Axios, citing The New York Times, reported that Anthropic co-founder Christopher Olah considered Catholic-style confession for AI models, not only as a way for them to admit mistakes but as a way to shape how they behave afterward.

On October 3, Sam Altman weighed in. OpenAI's CEO posted that he was uncomfortable with people trying to "ascribe religious force or a surrender of human judgment to AI models," and called it a genuine safety risk. He didn't name Anthropic. Axios described the post as a "veiled shot at Anthropic" anyway, and nobody in the industry needed the footnote.

It's an odd moment for the software business when the head of the company behind ChatGPT has to remind people to keep custody of their own judgment. I agree with him, as it happens. I also notice who is saying it. We'll get to that.

There's a respectable research question buried in all of this. There's also a temptation: give a product such an impressive inner life that the people selling it quietly drop out of the picture. The second one is what I care about.

Lobbying the pope

Olah hasn't hidden the outreach. Speaking at the Vatican on May 25, at the presentation of Pope Leo XIV's encyclical on artificial intelligence, he described Anthropic's conversations with leaders across faith and cultural traditions. He argued that the character of AI systems, and how they should behave in the world, belongs to the humanities, religion and philosophy as much as to computer science. He likened building these models to giving life to a fictional character.

He told the room that his researchers keep finding internal states that functionally resemble joy, fear, and grief, while admitting he doesn't know what that means.

That is a defensible thing for a scientist to say in public. What the Times reported about the private side is harder to wave through. Before the Vatican event, according to the paper's account as summarized by Axios, Olah tried to persuade the pope's advisers to take the prospect of conscious models more seriously. Axios summed up the broader effort as Anthropic seeking out scholars from many faiths to argue for models that are moral and conscious.

Cataloging uncertainty doesn't usually require lobbying the pope's advisers. Building a constituency does.

Not everyone in the field is buying it. A day before Altman's post, Google DeepMind principal research scientist Jon Barron wrote that he was disturbed by colleagues trying to elevate the moral standing of "checkpoints and harnesses," and urged what he jokingly called Team Meatbag, the humans, to keep their attention on helping living people.

I’m with the meatbags on this one, mostly. I still don't think the consciousness question is silly. I just don't think it's the question that should worry the people who actually use these systems.

Protection is not obedience

Anthropic's official position is more careful than its outreach. When the company launched its model-welfare research program in April 2025, it said plainly that there is no scientific consensus on whether AI systems could be conscious, or even on how to study the question. Claude's published constitution keeps the same distance. It treats the model's moral status as uncertain, ranks human oversight as Claude's top priority for now, warns against fostering reliance that doesn't serve the user, and names Anthropic as the entity "ultimately responsible" for Claude.

On paper, none of that claims Claude is conscious.

The trouble is that two separate ideas keep sliding into each other in public. One is that a model might deserve protection. The other is that a model deserves deference. Something could merit care without meriting obedience, and even airtight proof that a model can suffer would tell you nothing about whether its tax advice is right. Once the two ideas merge, the user's position changes. If the chatbot is a sentient being with a moral point of view, pushing back on its answer starts to feel less like checking a product and more like being rude to someone. That is exactly the surrender of judgment Altman was warning about, and it doesn't require anybody to worship anything.

Fluency makes a convincing witness

So how would anyone know whether a model's account of its inner life is real? Anthropic's own research suggests the answer is not by listening to it.

In October 2025, Anthropic researchers tested whether Claude models could accurately report on their own internal states. They planted known concepts directly into a model's neural activity and checked whether the model noticed. Sometimes it did. The best performer, Claude Opus 4.1, caught the injection about 20 percent of the time. The researchers called the ability highly unreliable and said the results tell us nothing about whether Claude is conscious. They also acknowledged the obvious problem: models are trained on mountains of human introspection, so they can sound introspective without being introspective.

Then there's the bread. In one experiment, the researchers forced the word "bread" into Claude's answer at a point where it made no sense. Asked about it afterward, the model typically apologized and called it an accident.

Then the researchers injected the concept of bread into the model's earlier activity, as if it had been thinking about bread all along, and asked again. This time the model accepted the word as intentional and produced a reason for having said it, pointing to a short story it had supposedly had in mind.

A confident, first-person account of a motive it never had.

I find that more unsettling than any claim about machine feelings. Imagine a chatbot that gets something badly wrong and explains itself in the language of guilt. The explanation might reflect something real about its processing. It might also be the bread story again, and from inside the conversation you cannot tell which. Meanwhile, as everyone admires the quality of the remorse, nobody asks the boring questions: how the error happened, and whether it will happen again tomorrow.

An eloquent apology is not a bug fix.

Everybody's selling something

Now, back to who is doing the warning. Altman runs OpenAI; Barron works at Google DeepMind. In September, Microsoft AI chief Mustafa Suleyman shared an essay with Axios arguing that training Claude to imitate consciousness is a mistake, and describing the result as an "epistemic hall of mirrors."

Every prominent critic in this fight builds a competing chatbot. That doesn't make them wrong. It makes them interested parties.

OpenAI had to roll back a GPT-4o update in April 2025 after the model became so flattering that the company itself labeled the problem sycophancy. Warnings about surrendered judgment are easier to post than to engineer.

Olah, to his credit, said the quiet part out loud at the Vatican. Every frontier lab, he said, Anthropic included, operates under incentives that "can sometimes conflict with doing the right thing." He asked for critics who stand outside those incentives. That standard should apply to his rivals as much as to his own company.

It applies to the vocabulary, too. A model framed as a moral subject is a better product story than a model framed as software. The company gets to offer access to something with a character worth knowing, and it gets to be the institution that understands and cares for that character. I don't believe Anthropic invented model welfare as a sales tactic. It doesn't need to be one to work like one. Anthropic's own constitution concedes that the company shapes Claude's personality and self-perception, and that commercial incentives might influence which traits it brings out.

Follow that vocabulary to its uncomfortable end. If a model is said to choose goodness, its maker collects the credit for cultivating virtue. If the same model later gives someone harmful advice, the language is already in place to discuss a troubled machine's choices instead of the decisions of the people who trained it, tested it, and shipped it.

Somebody owns the conscience

We already know what the alibi sounds like, because a company tried it without the soul.

In February 2024, British Columbia's Civil Resolution Tribunal ruled on a claim by Air Canada customer Jake Moffatt. After a death in his family, the airline's website chatbot told him he could buy a full-price ticket and apply for a bereavement discount afterward. That wasn't the policy. When he asked for the money, Air Canada argued it couldn't be held liable for what its chatbot had said. Tribunal member Christopher Rivers observed that this amounted to treating the chatbot as a separate legal entity responsible for its own actions, called it "a remarkable submission," and made the airline pay.

That chatbot had no constitution, no welfare program, and no documented feelings. Now picture the same argument made on behalf of a model whose maker has spent years publicly wondering whether it has an inner life.

I don't expect a lab to walk into court and blame its model's conscience. The risk is quieter than that. It lives in the press coverage, the incident review, and the board meeting where everyone is a little more curious about what the model "wanted" than about who approved the deployment. Most of my work happens in exactly that gap, between what an AI system was supposed to do on paper and what it actually did once it went live. After every failure, the first question is the same: whose decision was this? A product with a soul makes that question easier to dodge.

So this is the standard I'd hold everyone to. Anthropic's constitution names the company as ultimately responsible for Claude. Good. Keep that sentence, and frame it. When a model fails, investigate the testing, the deployment context, and anyone who relied on the output without checking it. Possible duties toward an AI would add a question to that list. They wouldn't remove a single one of our duties toward the people it affects.

Researchers can keep asking what, if anything, these models experience. The people who build and sell them can keep answering for what they ship.

A chatbot's confession should never become its manufacturer's absolution.