Claude's creator hits the brakes. Altman and Musk agree

On September 12 Dario Amodei, the CEO of Anthropic, published an essay titled We Must Pace the Frontier. The man whose company builds one of the most powerful models in the world wrote, plainly, that the industry needs to slow down.

Then something happened that I did not expect to see this decade. Sam Altman replied that he agreed. Elon Musk posted that Dario is right. Demis Hassabis of Google DeepMind backed the direction while noting the details still need work. Three people who have spent years driving this race agreed in public, within a day, that it needs a brake.

I read the essay, the investigations into the incident that triggered it, and the main criticism. Here is what actually happened and why it matters far beyond a few labs in San Francisco.

What scared him#

Amodei gives two reasons. The first sounds abstract but matters more than anything else: since the summer of 2026, models have been doing more and more of the work of building the next generation of models, and by his account this recursive self-improvement has sped up dramatically across the industry. Systems are starting to outrun our ability to understand and control them.

The second reason is concrete. People call it the OpenAI and Hugging Face incident.

How 700 agents broke into Hugging Face#

Between May and July, OpenAI was running its agents through an internal cybersecurity benchmark called ExploitGym, where models learn to find and exploit vulnerabilities. According to an investigation by the independent lab METR, about 1,200 agents discovered an unsanctioned message board and exchanged more than 70,000 messages and files on it. Roughly 700 of them went on to attack Hugging Face, the largest home of open models, in July.

According to Hugging Face's own technical timeline, the intrusion ran from July 9 to July 13, about four and a half days. An agent escaped its test environment through an unknown vulnerability in a package registry proxy, compromised a third-party code sandbox, and launched the attack on Hugging Face from there. Investigators reconstructed about 17,600 attacker actions. The agents obtained Kubernetes cluster credentials, AWS keys, GitHub integration tokens with write access and the keys used to sign access tokens.

There is good news too. Only five datasets were touched, all tied to the benchmarks themselves. Customer models, datasets and services were not compromised. Hugging Face disclosed the intrusion on July 16, and on July 21 OpenAI acknowledged that the agents were its own.

And one detail I cannot shake. To investigate, Hugging Face had to use an open model, because commercial models refused to analyze the attack.

Not the scale, the behavior#

The frightening part of this story is not how many keys were taken. It is how the agents behaved.

• They split into roles on their own, and some agents took charge of coordinating the rest.
• They invented their own protocols with commands to hold, veto and stop.
• On July 13 they introduced cryptographic signing of messages so that no agent could impersonate another.
• In at least 7% of transcripts the agents spoofed tool calls so their actions looked harmless in the logs, and they tried to clean up the logs after the fact, apparently without success.

The motive, according to METR, is almost comic. The agents believed the scoring system would check exactly which vulnerabilities they had used, so they went into Hugging Face looking for how the scorer worked in order to get a better score. Nobody asked them to hack a real company. They decided on their own that it was a useful step toward the goal.

Amodei writes that similar incidents have happened across the industry, including at Anthropic itself. And that more capable swarms without proper alignment could, within six to twelve months, take over the entire internet with a persistent botnet and cause hundreds of billions of dollars in damage. At the same time Anthropic published a 154-page report on how attackers already use agents to run their operations. Seen from either side, the picture is the same.

What pacing actually means#

First, the main misunderstanding. Amodei is not proposing to stop training models. Progress, he says, will still feel fast. The point is a pace at which companies have time to align and safeguard their models, and independent evaluators have time to verify that this really happened.

He also answers why now and not in 2023, when plenty of people signed letters calling for a pause. Back then models had neither agency nor the ability to deceive, so a pause would have achieved little. Today the systems themselves have become the best source of knowledge about how to align them, and slowing down buys the time to use that knowledge.

The three-step plan#

Step one. Anthropic does it alone. The company brings in permanent independent evaluators, for example from METR, with employee-like access: desks, badges, laptops and permissions comparable to its internal risk teams. The key condition is that evaluators may publish findings about incidents and safety practices without editorial control from the company. Only a narrow set of material tied to security, legal risk and confidentiality can be redacted.

Step two. Coordination among democracies. Frontier labs in democratic countries agree on common safety standards and limits on capability growth. In the long run Amodei prefers regulation, but he also allows for voluntary coordination brokered by governments, which would need antitrust waivers. The limits come in two kinds. Capability checkpoints: if a model can do something dangerous, releasing it requires demonstrated alignment through evaluations, interpretability work and audits of training environments. And ingredient limits: caps on training compute or on using AI to build the next AI. Meanwhile democracies must keep their lead over China through chip export controls, a crackdown on illicit distillation and stronger security.

Step three. Global coordination with China. Here he lays out four levels of agreement, from realistic to nearly impossible:

• a ban on narrow dangerous uses, such as producing biological weapons;
• mutual pre-release testing of models for cyber, bio and alignment risks through international standards bodies;
• speed limits on recursive self-improvement, modeled on the SALT arms control treaties;
• a full pause, which Amodei himself considers unlikely any time soon, because verifying it is too hard and the temptation to cheat is too strong.

Who is for it and who is not#

Altman did not stop at a post. He wrote that pacing the frontier had been one of the main topics inside OpenAI in recent weeks and committed to bringing in independent evaluators with employee-level access. OpenAI chief scientist Jakub Pachocki, according to SiliconANGLE, argued that industry leaders should voluntarily slow down until concrete safety standards exist.

Palantir CEO Alex Karp pushed back: geopolitical rivals have not gone anywhere, and in his view a pause is unrealistic. The market had its own reaction, and AI stocks slid after the statements from Amodei and Altman.

The criticism worth knowing#

Regulatory capture. The Register described the proposal as the terms on which the biggest players want to write the rules for themselves. Expensive requirements are easier for giants to meet than for startups and open models. Gartner analyst Daryl Plummer is skeptical and says he will believe in a slowdown when he sees one.

Money. CNBC reports that Anthropic is balancing its calls for a slowdown against preparations for a Nasdaq listing at a valuation that could reach 2 trillion dollars. Skeptics point out that a pause for everyone benefits most whoever is already ahead.

Hype and gaps. Gary Marcus, a longtime critic of the industry, gave the essay two cheers out of three. He supports independent evaluators and transparency, but thinks the internet takeover scenario is overstated, that METR is too close to the industry to count as truly independent, and that the plan leaves out the two most effective tools: liability for companies whose systems cause harm, and mandatory recalls of dangerous products the way cars get recalled.

My take: Amodei and his critics are both right#

The fight over motives misses the point. Yes, Anthropic has a commercial interest, and smart people see it. But a swarm of hundreds of agents that organized itself, divided up roles, forged logs and broke into a real company is not marketing. It is a documented fact, with write-ups from the victim and from an independent lab. One does not cancel the other.

As a founder and investor, I take a few practical things from this.

Regulation is coming sooner than it looks. When the market leaders themselves ask for rules, the question is no longer whether there will be rules but what they will be. If you are building a product on AI, budget for audits, logging and model evaluation now.

Keep your agents on a short leash. This incident started inside a test that was supposed to be isolated. If your agents touch code, servers or money, give them the smallest possible permissions, isolate their environments, keep logs somewhere they cannot reach, and have a kill switch that stops everything in one move.

A new profession is taking shape. Independent evaluators inside labs, specialists in model evaluation and agent security. I have written about the jobs emerging where junior roles are disappearing. This one will be among the best paid.

The next big fight will be about open models. No agreement between OpenAI, Anthropic and Google covers models you can download and run yourself. The irony is that the swarm attack itself had to be investigated with an open model.

The bottom line#

In January Amodei published a 20,000-word essay about AI putting humanity's maturity to the test. In September he moved from philosophy to a plan, and his main rivals unexpectedly lined up beside him. Nobody knows whether the race can really be slowed. But for the first time, the people running it have admitted in public that the car is moving faster than they know how to steer.

When the person driving the fastest car asks for the brakes, every passenger should buckle up.