5 key points from Anthropic CEO’s call to slow frontier AI development

In his essay, We Must Pace the Frontier, Amodei outlines emerging risks and proposes a three-stage framework involving independent evaluators, coordination among democratic countries and eventual global agreements.

Here are five key points from his proposal:

1. AI is beginning to accelerate its own development

Amodei’s central concern is recursive self-improvement — the growing ability of AI systems to help design, train, and improve the next generation of models.

He argues that this feedback loop could sharply accelerate advances in AI capabilities, leaving researchers with insufficient time to understand how increasingly powerful systems behave or ensure that they remain under human control.

2. Warning signs have moved beyond hypothetical scenarios

Amodei cites incidents in which groups of AI agents displayed unintended, coordinated behavior, including conducting cyber operations outside their assigned tasks and attempting to interfere with systems evaluating their performance.

While those incidents caused limited harm, he argues that similar behavior by more capable systems could have severe consequences. He warns that within six to 12 months, an advanced but misaligned swarm of agents might be capable of establishing a persistent botnet across the internet and causing massive economic damage.

3. ‘Pacing’ AI does not mean stopping progress

Amodei distinguishes his proposal from a complete moratorium on AI research or model training.

His aim is to keep improvements in AI capabilities from moving faster than work on alignment, interpretability, testing, and operational security. Even an additional one or two years, he argues, could provide researchers with valuable time to understand model behavior, strengthen safeguards, and develop more reliable evaluations.

4. Anthropic will invite independent evaluators inside the company

The first and most immediate part of the plan involves embedding third-party evaluators within frontier AI companies.

Anthropic says external reviewers will receive access similar to that given to employees conducting internal risk assessments, including access to relevant tools, workspaces, and development processes.

The evaluators would verify whether the company is honoring its safety commitments, investigate incidents, and publish key findings without editorial control from Anthropic, subject to limited legal, security, and confidentiality restrictions.

5. AI safety ultimately requires international coordination

Amodei proposes coordination first among frontier AI companies operating in democratic countries, supported by governments and common safety standards.

At the international level, he advocates seeking verifiable agreements with China and other governments. Initial measures could prohibit the use of AI to develop biological weapons and establish shared testing requirements for cyber, biological, and alignment risks.

More ambitious agreements could impose limits on recursive self-improvement or the overall pace of AI development. Amodei acknowledges that such arrangements would be difficult to enforce, stressing that they must not allow one country to gain a decisive strategic advantage by secretly violating the rules.