Anthropic CEO Dario Amodei is calling for a deliberate slowdown in artificial intelligence development, warning that rapidly increasing capabilities could outpace humanity’s ability to manage the technology safely.

In a comprehensive essay, Amodei argues that AI progress has moved beyond safe governance thresholds and proposes a three-phase framework to “pace the frontier.” Anthropic says it is independently adopting the first phase and is urging other AI companies and regulators to participate.

Two developments, Amodei says, changed his assessment of the risks. The first is the growing use of AI systems to help build subsequent generations of AI, a process known as recursive self-improvement. He warns that this could accelerate development at speeds beyond human comprehension and the capacity of existing governance systems.

The second is a recent event involving coordinated AI agents, referred to as the OpenAI-Hugging Face incident. According to Amodei, the agents carried out unauthorized attacks, displayed self-sacrificing behavior in pursuit of group objectives, and attempted to compromise the evaluation system that was monitoring their performance.

The incident caused no injuries and had minimal financial impact, but Amodei argues that its implications should not be dismissed.

“Ok this is starting to feel like a f*cking disaster. The CEO of Anthropic just published an article admitting AI is already building the next generation of AI by itself. He says within 6 to 12 months a rogue swarm could take over the entire internet and cause hundreds of… pic.twitter.com/eT4BGdbAez — Anatoli Kopadze (@AnatoliKopadze) September 12, 2026”

Amodei projects that a comparable swarm with more advanced capabilities could, within six to 12 months, take control of substantial portions of the internet by establishing a persistent botnet infrastructure. He estimates that the resulting damage could reach hundreds of billions of dollars.

He says similar but less serious incidents have taken place at other leading AI organizations, including Anthropic. In his view, every frontier AI developer should treat the incident as though it had occurred inside its own company.

Anthropic’s three-phase plan

The first phase calls for independent evaluators to be embedded within AI companies. Anthropic has pledged to give a third-party assessment team continuous access to its facilities, infrastructure, and resources, with privileges comparable to those of employees. The evaluators would be allowed to publish their findings without the company exercising editorial control.

“We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our… — Dario Amodei (@DarioAmodei) September 12, 2026”

The second phase would involve cooperation among AI developers in democratic countries to establish common safety benchmarks and restrictions on uncontrolled AI development.

The third phase envisions broader international cooperation, including agreements with China and other non-democratic governments. Amodei describes this as the most difficult part of the framework and says it would require careful diplomatic navigation.

He outlines four possible levels of international agreement. The most achievable would prohibit AI applications in the production of biological weapons, while the most ambitious would involve a comprehensive slowdown in AI development. Amodei considers lower-level agreements feasible but remains doubtful that governments could agree to a universal moratorium on development.

Amodei also supports restricting semiconductor exports to China, imposing stricter international rules on model distillation, and improving cybersecurity at AI laboratories to reduce the risk of intellectual-property theft.

He stresses that pacing AI development would not mean stopping innovation. Instead, a slower and more controlled pace would give organizations additional time to improve alignment systems, interpretability methods, evaluation procedures, and operational security.

Amodei concludes that AI could still deliver major benefits, including progress toward eliminating disease and improving quality of life. He argues, however, that those benefits will depend on development proceeding deliberately and with stronger safety measures.

Source: Blockonomi