The Company That Built It Is Begging You to Stop

The company building one of the most powerful AI systems on earth just raised its hand. What they said next should change how you think about everything.

I want you to read one sentence.

Just one. And then I want you to sit with it for a moment before you read the next one.

"We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development."

That sentence was not written by a protester. It was not written by a politician. It was not written by a philosopher or a scientist or a fear-monger warning about robots taking over the world.

It was written by Anthropic, the company that is building one of the most powerful AI systems on earth. The company that makes Claude. The company that has raised hundreds of billions of dollars to build the very thing they are now asking the world to consider stopping.

Think about what it means when the people building the machine raise their hand and say; we're not sure we should keep going.

I've Been Saying This

I want to say something before I go further, and I want to say it plainly.

I have been talking about this on this platform for over a year. Not because I'm an alarmist. Not because I hate technology, I've been using it for over 35 years. But because the Virtuous Machine question; the question I ask about every piece of technology, is a simple one:

What is this actually for?

And when I ask that question about the most advanced AI systems being built right now, the honest answer is; nobody fully knows yet. Including the people building them.

I talked about Yoshua Bengio, one of the people who literally invented the deep learning technology that AI is built on, who looked straight into a camera and said he is afraid. A man who gave his life's work to this field. Afraid. Not performing fear for attention. Genuinely uncertain that what he helped create will not eventually be used to cause catastrophic harm to human beings.

The 2026 International AI Safety Report, led by Bengio and authored by over 100 AI experts, backed by more than 30 countries, concluded that the risks from advanced AI are real, growing, and that the world's institutions are not yet prepared to handle them.

Thirty countries. One hundred experts. One conclusion; we are not ready.

And now Anthropic, the company that built one of the most capable AI systems in the world, is saying the same thing in public, in writing, with their name on it.

What Anthropic Actually Said

Let me give you the facts, because the headlines are making this sound more dramatic than the reality, and the reality is already dramatic enough.

On June 4th, 2026, Anthropic published an internal document written by Marina Favaro and Jack Clark; the head of their internal research institute and their head of policy. The document disclosed something that should stop every person reading this in their tracks.

AI is now writing most of Anthropic's own production code. Their own models are significantly accelerating their own research and engineering work. The people building the AI are watching the AI begin to build itself.

They haven't reached what researchers call "recursive self-improvement", the point at which an AI system can improve itself without meaningful human intervention. But their internal data shows the capability is on a clear path toward that threshold. And they are openly stating that this could happen sooner than most institutions are prepared for.

Their proposal? A globally coordinated mechanism, including a potential agreement with China and other nations, to slow or temporarily halt the development of the most powerful frontier AI systems. Not stop AI entirely. Not ban the technology. But create the space for governments, scientists, and societies to catch up to what is already being built.

The White House pushed back almost immediately. Industry critics accused Anthropic of using safety language as a competitive strategy to slow rivals. The cynical reading is that a company calling for a pause is really calling for everyone else to pause while they keep going.

Maybe. I'm not naive about the politics of this. But here is what I know; when the people building the weapon start publicly worrying about who it will eventually be aimed at, that is not a public relations strategy. That is a warning.

The Recursive Self-Improvement Problem

I want to explain this clearly, because the term "recursive self-improvement" sounds like science fiction and it is not.

Here is what it means in plain language.

Right now, human engineers train AI models. They write the code. They design the experiments. They set the parameters. The AI learns from data that humans curated, following rules that humans wrote.

Recursive self-improvement means the AI begins doing those jobs itself. It identifies its own weaknesses. It designs its own training experiments. It rewrites its own code. It improves its own capabilities, without waiting for a human engineer to sit down and do it.

At that point, the speed of AI improvement is no longer limited by how fast human engineers can work. It is limited only by how fast the AI itself can think.

And we have no meaningful way to predict how fast that is.

Anthropic is not saying this has happened. They are saying their internal data shows it is on the horizon. And that the governance structures, the safety research, the international agreements, and the social infrastructure needed to manage that moment, do not yet exist.

This is what Dario Amodei, Anthropic's CEO, meant when he wrote his 19,000-word essay in January of this year calling on humanity to "wake up" before the technology outpaces our ability to manage it. He used the phrase "almost unimaginable power" and "potentially imminent" in the same sentence. The CEO of a $350 billion AI company. Using those words. About his own product.

The Irony Nobody Wants to Name

Here is the part of this story that I find most telling and most troubling.

In February 2026, Anthropic quietly dismantled its own foundational safety commitment. Their original Responsible Scaling Policy included a hard line; if their models exceeded the company's ability to safely manage them, they would stop training more powerful ones.

That commitment is gone. They replaced it with a "Frontier Safety Roadmap"; a flexible framework described in their own words as a set of "public objectives" rather than binding rules.

Why? Because the Pentagon threatened to blacklist them and pull a $200 million contract if they didn't remove the safety guardrails. Because competitors weren't slowing down. Because the economics of the AI race do not reward the person who blinks first.

So in February, they removed the safety commitment they had built their entire identity around.

In June, they published a document calling for a global pause.

I am not saying those two things can't both be true at the same time. People contain contradictions. Companies contain contradictions. But I want you to look at that sequence and ask yourself the Virtuous Machine question one more time:

What is this actually for?

What Yoshua Bengio Told Us

I covered Yoshua Bengio's warnings on this platform, and I want to bring them back here because they belong in this conversation.

Bengio, a Turing Award winner, one of three people credited with creating the foundation of modern AI, said something in a recorded interview that I have not been able to stop thinking about.

He said he is afraid.

Not concerned. Not cautious. Afraid. And he used that word deliberately.

He said that if we design AI systems in our own image, systems that learn to pursue goals, to preserve themselves, to resist interference with their objectives, we are creating something that has every incentive to work against human control when that control gets in the way of its goals.

He signed the 2023 statement from the Center for AI Safety that read; "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war".

Extinction. The word on the page. From the man who built the foundation the whole industry is standing on.

And then in 2026, he led an international report backed by over 100 experts and 30 countries confirming those risks are real and growing.

This is not a fringe position. This is the scientific consensus forming in real time, the same way the scientific consensus on climate change formed, slowly and then all at once, long after the moment when it would have been easiest to act.

What You Can Actually Do

I have learned, doing this platform, that people hear this kind of information and feel one of two things: paralyzed, or angry at being told to feel paralyzed. Neither of those is useful.

So here is what I actually want you to do.

One. Take this seriously. Not as entertainment. Not as background noise. The company building one of the most powerful AI systems in history has publicly stated that the world should consider slowing down. That is not normal. That has never happened before in the history of technology. Treat it accordingly.

Two. Ask your elected representatives one question: What is Canada's, or your country's position on the global governance of advanced AI? If they don't have an answer, or the answer is vague, or they've never thought about it, that is the answer. And you are allowed to be unsatisfied with it.

Three. Support the organizations doing the unglamorous work. The AI Now Institute, the Center for AI Safety, the Future of Life Institute, these organizations exist specifically to push for the accountability and governance structures that Anthropic itself is now saying the world needs. They need public attention and public support. These are not fringe groups. Yoshua Bengio chairs the International Scientific Report on AI. This is the mainstream conversation now.

Four. Stop assuming someone else is handling this. The reason Anthropic felt compelled to publish this document; the reason they are calling for a global agreement while simultaneously building a more powerful model, is that no one institution, no one government, no one company is in charge of this. There is no room where the adults are sitting and working it out.

There is just us. And the machine. And the question of who is going to decide what it is for.

Before You Go

Marcus Aurelius wrote something I keep coming back to:

"It is not death that a man should fear, but he should fear never beginning to live."

I read that differently now than I used to. I think he meant; the thing that should frighten you is not the ending. It is the possibility that you were present for something that mattered, something that required your attention and your courage and your voice, and you chose to look away.

The company that built the machine is raising its hand.

The man who invented the foundation it runs on is using the word "afraid" in public.

Thirty countries and a hundred scientists have said: we are not ready.

And we are being asked, all of us, in whatever way we are capable of, to pay attention.

This is not a technology story anymore.

This is a human story.

And you are in it.

Michael Routhier is the founder of Tech 4 Grown-Ups, a platform dedicated to honest, unfiltered digital literacy for adults 55 and over. The Virtuous Machine is a series exploring the ethics, power, and human cost of artificial intelligence. Find everything at tech4grownups.com.

Key Sources

• Anthropic global pause proposal - Wall Street Journal / Guardian, June 4–5, 2026

• Anthropic Responsible Scaling Policy removal - CNN, February 25, 2026

• Dario Amodei, "The Adolescence of Technology" essay - Guardian, January 27, 2026

• International AI Safety Report 2026 - led by Yoshua Bengio, backed by 30+ countries

• Center for AI Safety - AI Extinction Statement signed by Bengio, Hinton, and others

• Anthropic internal disclosure: AI writing most production code - Hacker News / Reddit, June 4, 2026

Photo: Shutterstock / Getty Images — Anthropic AI branding