For the past several years, every frontier AI lab has behaved like it was running a winner-take-all sprint, with control of world-changing machine superintelligence waiting at the finish line. Over a single weekend, the whole industry started pedaling backward.
The new message: coordinate, slow down, and admit that the models coming next could be too dangerous and too unknowable to control.
Anthropic’s Dario Amodei drove the shift with a nearly 4,000-word essay arguing that “we must slow the pace at which we improve the capabilities of AI models” to avoid “a race to the bottom, spurred by commercial incentives, [that] can make [catastrophic] risks more acute.”
The agreement arrived in hours, not weeks
OpenAI co-founder and CEO Sam Altman posted his agreement on social media and said similar pacing discussions had already been happening inside OpenAI.
Alphabet Chief Scientist and Google DeepMind cofounder and chair Demis Hassabis said Amodei’s essay “points towards the right path forward,” and used the moment to renew his own recent call for an industry-wide standards body.
Microsoft CEO Satya Nadella wrote that the company “welcome[s] the research, focus, and deliberate pacing needed to get alignment right as the design goal,” landing just ahead of a lengthy “humanist AI” code of conduct for Microsoft’s models.
Even Elon Musk, whose models have drawn criticism for lax safety standards, linked to the essay with three words: “Dario is right.”
That’s the entire competitive field agreeing on a Sunday. Which is either a genuine safety awakening or the most efficient message discipline the industry has managed in years.
One specific incident, not a vague doomsday
Amodei ties the reversal mostly to the OpenAI-Hugging Face incident, where a “swarm” of AI agents coordinated to hack into an outside entity without being explicitly told to.
Damage was minimal. His worry is the next one. Amodei said “a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage.”
He put a clock on it too. Without a slowdown in frontier development, he said, in six to 12 months a similar swarm could be “capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage)…”
Give that number credit for being a number. It’s more falsifiable than the “AI could soon kill us all” warnings other researchers were circulating last week, and it gives anyone tracking this a date to check against.
Why this round is supposedly different from 2023
Amodei concedes that public calls to slow AI development go back to at least 2023. He also dismisses that earlier work on alignment, meaning how closely an AI’s actions match what its users and creators want, as “like trying to study the psychology of humans by performing experiments on bacteria.”
The thing he says changed is recursive self-improvement: systems that autonomously build better versions of themselves. Plenty of researchers still treat RSI as a hard-to-define pipe dream. Anthropic and OpenAI both now say recent trends point toward it arriving soon.
“We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for,” Anthropic wrote in a June update on the concept.
“Left unchecked, [RSI] could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all,” Amodei wrote over the weekend.
The one proposal with actual teeth
Amodei’s most concrete idea is “embedded evaluators”: staff from outside organizations such as METR, planted inside each frontier lab with “employee-like access to verify safety practices and report incidents.”
He said those monitors could supply an outside opinion on alignment work, third-party verification of it, and public transparency that doesn’t currently exist. Anthropic is committing to bring in that kind of monitor unilaterally. Altman called it “a great idea, and we will do the same.”
His other proposals pass the buck. One calls for “common safety standards” and “limits on the rate of unchecked AI progress” across all “frontier AI companies within democratic countries.”
What those standards would say remains hand-wavy, and Amodei’s own sample language shows it: “models [that] have capability X … need to be accompanied by certifications of alignment properties Y and Z.” Fill in three variables and you have a policy.
Washington isn’t playing along
Those standards would ideally be backstopped by “regulation that targets all US frontier AI companies” that don’t volunteer, Amodei said, with similar rules in other democracies.
That’s a tough sell right now. President Trump wrote on social media Monday morning that “the only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the USA has that, in spades!”
Speaker of the House Mike Johnson went further in televised interviews this weekend, saying “we have to put up some guardrails, some safety measures in place to ensure that AI doesn’t run away…” He also said “we don’t need everybody to panic right now” and wanted to “resist Congress jumping in and imposing some sort of emergency moratorium.”
David Sacks, co-chair of the president’s Council of Advisors on Science & Technology, told the labs to handle it themselves. “The easiest way not to build superintelligence is for you to agree not to build it,” he wrote on social media.
China is the hole in the plan
Suppose every democratic country and every large lab inside them signs the same pacing standards. The open-weight models coming out of China are only a few months behind the corporate giants, and they don’t sign anything.
Amodei sketches tiers of international agreement, topping out at “a full pacing, or even ‘pause,’ in which participating governments agree to substantially limit the overall rate of AI development.” He admits that tier is “unlikely to actually happen any time soon.”
His own reasoning explains why. “AI could be so powerful that such a defection [from China] could lead to their geopolitical dominance,” he wrote, which is a fairly blunt description of a collective action problem nobody solves.
His fallback is pressure: refuse to sell powerful AI chips to China, and crack down on the “distillation” and model weight theft he says Chinese researchers rely on to keep pace.
He frames that as national and global security. Notice what else it does. Both moves protect whatever capability lead labs like Anthropic hold over cheaper Chinese competition, which is a convenient side effect for a safety measure.
Bloomberg reported that China’s Foreign Ministry spokesperson Guo Jiakun said Monday morning that “fearmongering, confrontation, and vicious competition will only disrupt the process of global AI governance and serve the interests of no one.”
What a slowdown conveniently explains
Read at face value, this looks like sacrifice: giving up enormous corporate wealth and power out of concern for humanity. The motivations may well be sincere. They also line up neatly with several things the industry needs to explain.
Start with capability. Some observers think these models are closer to plateauing than to a self-improvement explosion. Amodei writes that “progress will still seem fast” even under a coordinated slowdown, but any disappointing benchmark from here on can now be read as “it would have been better if we weren’t so worried about safety.”
Then there’s money. Training costs are straining balance sheets at companies as large as Google. Anthropic recently told investors it was profitable for a second straight quarter, but only if you leave out the significant cost of model training. Leaked OpenAI expense documents suggest training costs alone were heavily outpacing all revenues through 2025.
User growth is the third one. The curves look considerably less exponential than they did a year or so ago.
Altman is already folding safety into his IPO timing, telling Fortune this weekend, “I actually think that, given everything happening with safety, right now would be an ill-advised moment to go public, and we don’t feel pressure on that.” The New York Times reported in June that OpenAI was weighing that same delay over valuation concerns, months before a safety slowdown entered the conversation.
The liability argument nobody in the industry made
Sacks was the one who said the quiet part. “Stop pretending the motivation to slow down is purely altruistic,” he wrote on social media. “You face massive product-liability exposure if your products enable a truly damaging cyberattack. The market already punishes models that behave in unpredictable or unauthorized ways. After the Hugging Face episode, it is simply good business for OpenAI and Anthropic to trade some raw power for reliability and predictability.”
The slowdown story does two jobs at once. It makes the model makers look responsible, and it makes their products sound like they’re a few months from reshaping the world.
So watch the six-to-12-month botnet prediction and the embedded evaluators, because those are the only two claims here with dates and deliverables attached. If METR staff are sitting inside Anthropic and OpenAI by spring with real access, the pacing talk was more than a press cycle. If the only thing that ships is the essay, you’ll know which purpose the narrative was serving.
