NewsMacroAnthropic warns AI-built AI could outrun safety oversight and tilt markets

Anthropic warns AI-built AI could outrun safety oversight and tilt markets

Author: Cryptopolitan·

Key Takeaways

  • •Claude produced more than 80% of the code engineers contributed to Anthropic's software projects by May 2026, and a typical engineer contributed eight times more code per day in Q2 2026 than in 2024.
  • •Anthropic's August measurements found Claude led 26% of AI research and development tasks, up from less than 1% in February, though none of the measured work was fully autonomous.
  • •Anthropic's June 4 evaluation warned that AI-accelerated development could surpass safety checks and regulatory oversight even before any system achieves autonomy.
  • •Anthropic CEO Dario Amodei called in a September essay for slowing AI capability improvements and creating independent evaluators, after OpenAI's September 16 disclosure detailed six cases of concerning model behavior.
  • •Private AI investment rose 127.5% in 2025, with US investment reaching 23 times China's, and concentrated access to chips, cloud infrastructure, and data could extend leading laboratories' advantage over smaller competitors.
Anthropic warns AI-built AI could outrun safety oversight and tilt markets

Artificial intelligence is taking on a growing share of the work required to build more capable models. Anthropic says Claude is already making that process move faster, even though the model still cannot create its own successor. In a June 4 evaluation, the company warned that this advancement creates a risk of AI directly surpassing safety checks and regulatory oversight.

When AI builds itself, and what it does not yet mean

Claude has not yet reached recursive self-improvement — the point at which an AI would create and train its own successor without any human input. According to Anthropic, no system has achieved that capability so far. The distinction matters for oversight, because it means the acceleration now underway is playing out inside human-supervised workflows, in the window before any system approaches autonomy.

Even without full autonomy, Claude has come to handle much of Anthropic's coding effort. By May 2026, the model had produced more than 80% of the code that engineers contributed to the company's software projects. In the second quarter of 2026, a typical engineer contributed eight times more code per day than in 2024.

Those figures show how much coding output has grown, though not necessarily how much useful work is getting done. A survey cited by METR, covering 349 technical workers, found a median self-reported increase of 1.4–2 times in the value of their work when using AI tools.

26% led by Claude, none of it fully autonomous

Claude's role now extends well beyond writing code. Anthropic's August measurements found that the model led 26% of AI research and development tasks, up from less than 1% in February. Claude collaborated on or led more than 90% of that work, but humans remained involved throughout, and none of the measured work was fully autonomous. That February-to-August jump also gives observers a baseline: future measurement updates offer a way to track whether AI's share of research work keeps climbing.

The two bodies of research measure different things. Anthropic tracks code generation and research carried out with the help of AI systems, while METR captures how workers perceive increases in the value of their work. Neither study indicates whether AI is capable of deciding which research problems should be solved in the first place.

How acceleration could compound without a self-improving model

AI does not have to be fully autonomous to speed up research. According to an October RAND report, AI agents can help create stronger models without ever building their own successors. Their access to sensitive models and code already poses a security risk.

Economists at METR note that factors such as the availability of data, computing power, and the capability to conduct experimental studies could impose limits. Those same factors give researchers concrete conditions to monitor when weighing whether acceleration will persist. Even so, they have not dismissed the possibility of a sustained acceleration in AI development. The open question is whether independent safety reviews can keep up with AI-powered experiments — otherwise, supervision could lag long before full autonomy arrives.

From pacing the frontier to policing it

That concern prompted Anthropic CEO Dario Amodei to call for slower development and independent evaluators. “We must slow the pace at which we improve the capabilities of AI models,” he wrote in a September essay.

OpenAI's September 16 disclosure detailed six cases of concerning model behavior. Cryptopolitan has also reported divisions among technology leaders over whether AI development should slow down.

But agreeing on safety concerns does not guarantee action. CSIS warned that outside evaluators must avoid conflicts of interest, and that governments must guard against regulatory capture. Industry-wide agreements remain uncertain, and whether independent evaluators take shape at all stands as the nearest-term test of whether the pacing debate produces concrete change.

The money follows the compute

The debate also carries substantial financial stakes. Stanford's 2026 AI Index found that private AI investment rose 127.5% in 2025, with US investment reaching 23 times China's. The OECD separately warns that access to chips, cloud infrastructure, and data remains concentrated.

The result is a paradox. AI may make research coding cheaper, but training and testing advanced models still requires enormous investment. Faster development could therefore give leading laboratories an even greater advantage over smaller competitors.