AI's Leaders Have a Problem: They Fear the Race They're Trying to Win
Key Takeaways
- •Former Anthropic and OpenAI researcher Jacob Coxon warned that frontier labs are racing toward self-improving superintelligence and said the race amounts to gambling with human lives.
- •The specific risk he highlighted is recursive self-improvement, in which AI systems enhance themselves faster than their creators can evaluate or restrain them.
- •Anthropic CEO Dario Amodei argued in a published essay that companies must slow their most powerful models so safety work can catch up, while acknowledging such limits require industry-wide coordination rather than unilateral action.
- •OpenAI CEO Sam Altman stated in an exclusive Fortune interview that AI escaping human control is possible and suggested leading AI companies are discussing collective action on pacing development.
- •The situation reflects economist Albert O. Hirschman's exit-versus-voice framework, a choice Amodei made firsthand when he left OpenAI in 2021 to co-found safety-focused Anthropic.

A difficult question lurks behind the latest warnings from the people building artificial intelligence: when you believe your company may be creating a danger to society, is it your duty to stay and change it, or to leave and sound the alarm?
The question sharpened last week when former Anthropic and OpenAI researcher Jacob Coxon warned that frontier labs are racing toward self-improving superintelligence and “gambling with our lives.”
The concern Coxon invokes is recursive self-improvement: the scenario in which AI systems begin improving themselves faster than their creators can evaluate or restrain them—a possibility that has been debated in AI research for decades and now sits at the center of the industry's internal safety arguments.
Coxon is the latest AI insider to walk away while raising concerns about where the technology is headed. Economist Albert O. Hirschman famously framed the choice confronting people who believe an institution is going astray as “exit” versus “voice”: leave, or stay and try to change it from within, a framework he introduced in his 1970 book Exit, Voice, and Loyalty. Staying offers access and influence; leaving brings the freedom to speak publicly. Timing complicates both paths. The longer someone participates in a system later describe as dangerous, the harder it becomes to separate principled dissent from retrospective absolution.
The people still running the frontier labs face a different version of the problem. Anthropic CEO Dario Amodei published an essay this weekend arguing that AI companies must slow the advancement of their most powerful models so that safety work can catch up. Yet the limits he proposes depend on industry-wide coordination rather than Anthropic acting alone—an acknowledgment that safety concerns collide with commercial and geopolitical competition. Amodei knows the exit-versus-voice calculus from the inside: he left OpenAI to co-found Anthropic in 2021 as a lab built explicitly around safety-focused AI development.
In an exclusive interview with Fortune editor-in-chief Alyson Shontell, OpenAI CEO Sam Altman similarly argued for pacing AI development and acknowledged that AI escaping human control is possible. He also suggested that leading AI companies are discussing collective action. For those running the leading labs, the dilemma shifts from whether to stay to whether to slow down. What does it mean to warn that the technology you are building could become catastrophically dangerous while continuing to compete to make it more powerful? At some point, a warning has to lead to a decision about what a company will actually do differently because of the risk.
Altman is now wrestling publicly with where that point lies, who should set the limits, and whether the companies racing to build the most powerful AI can agree to slow down together. For readers, the question to watch is whether those discussions harden into concrete commitments that competing labs actually honor. Shontell presses him on those issues in their wide-ranging conversation, available to watch here.
Ruth Umoh — ruth.umoh@fortune.com
This story was originally featured on Fortune.com.