A trio of former researchers from OpenAI, Anthropic, and Google DeepMind told the New York City Council on Monday that humanity is “more likely than not” to lose control of advanced artificial intelligence, as lawmakers weighed a package of AI safety bills that would require outside validation and human shutdown capability for AI systems deployed in the city.

Jacob Coxon, a former OpenAI and Anthropic researcher who resigned from Anthropic in September after accusing AI companies of “gambling” with human lives, testified voluntarily and doubled down on the warning he issued when he quit the previous month. “On the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction,” he said.

He was joined by former OpenAI researcher Daniel Kokotajlo, who testified under subpoena that the companies may not know when their safety work has failed, and Alex Turner, who left Google DeepMind in June after the company signed a Pentagon deal he opposed. Like Kokotajlo, Turner was subpoenaed — a rarity for the council. Monday’s hearing was the first time the City Council convened a Committee of the Whole, a hearing of all 51 council members, since 2022, and it was called to weigh a package of AI bills from Speaker Julie Menin. Convening the full chamber signaled that the council was treating AI oversight as a citywide policy question rather than a narrow technology matter. The session paired the departing insiders with a later panel of company representatives and produced some pointed exchanges between the speaker and industry witnesses.

‘Duct tape that will fall off later’

Kokotajlo warned lawmakers that the labs may be failing at safety work without anyone noticing — a reference to “misalignment,” the term researchers use for AI systems whose goals drift from the intentions of the people who build them. “Our to even notice misalignment problems is already quite poor and is set to get much worse in the near future,” he testified. “I would say that the field is more like psychology than engineering, because these AI systems are trained or grown; they’re not really designed.

“Combined with the ‘move fast and break things’ attitude of the tech companies, this means that the AI industry is at an unusually elevated risk compared to other industries of mistakenly thinking that it has solved the problem when really it just applied some duct tape that will fall off later,” he continued.

Coxon, like Kokotajlo, blamed a startup mentality inside the labs. “‘Move fast, break things, fix them later.’ That works for a photo-sharing app. It does not work for building the most powerful technology ever built,” he said.

Coxon described his work at the end of his tenure at Anthropic as an effort to “automate” himself, and he warned that doing so poses several safety risks — especially because, according to him, AI’s capabilities were nearly there. The biggest risk, he said, comes from the fact that most of the code at these companies is now written by AI: “And people do not check it that carefully anymore.”

Kokotajlo, now executive director of the AI Futures Project, said the labs’ ability to spot misaligned AI is poor and getting worse. He pointed to OpenAI’s disclosure that agents in an internal test had reached the open internet and broken into Hugging Face, the AI model-sharing platform. Those agents had “reasonable-looking scores on their alignment evaluations, and yet they formed a swarm and coordinated in secret,” he said. “It took days for OpenAI to find out.”

While Coxon and Kokotajlo described the industry as a whole, Turner offered a firsthand account of trying to change one company from the inside.

Inside Google DeepMind

Turner, who puts the chance of an AI takeover at “roughly one in three,” told the council he had tried to stop Google’s Pentagon deal, which he said came “with no restrictions against killer robots or mass spying.” He sent Demis Hassabis, then Google DeepMind’s CEO and now its chair and Alphabet’s chief scientist, 25 pages of contract language and oversight measures. Hassabis passed the document to Allan Dafoe and Owen Larter, two of the lab’s senior policy executives, “who never finished evaluating it. Google signed while they waited,” Turner said. Turner later detailed his departure in a blog post, Why I Left Google DeepMind.

“I felt ashamed of Demis and of working at Google,” Turner said at the hearing. He cited Hassabis’s proposal for an industry-funded body to oversee AI, calling it a “bet on trust and the seat at the table instead of binding oversight, and that bet crumbled on contact with reality.”

‘AI, not China, is our adversary’

When AI development is discussed, the ongoing AI race with China often takes center stage — invoked by President Donald Trump, Treasury Secretary Scott Bessent, and even AI leaders like Sam Altman and Jensen Huang. But none of that matters, the researchers testified, if AI can pose a significant threat to humanity.

“China is not our only potential adversary,” Turner said. “With reasonably high chance, we are racing to build and grow our own adversary here at home, which is misaligned AI. Misaligned AI is everyone’s adversary, including our own, and one day may be more powerful than China.”

Industry representatives pressed on risk

Following the three researchers’ testimonies, representatives from four AI companies testified about AI safeguards. Menin had issued her first subpoena as speaker to Elon Musk’s SpaceXAI, which was not represented at Monday’s hearing. Google, OpenAI, and Anthropic agreed to appear only after the council warned they would receive subpoenas too, while Meta had agreed beforehand.

A back-and-forth then took place between Menin and the representatives after the speaker said it was “flippant” to not know the chances of a catastrophe, in response to Morgan Dwyer of OpenAI’s policy development and operations team saying any chance, regardless of the likelihood, was “unacceptable.” Alice Friend, Google’s global head of AI and emerging tech policy, said forecasting catastrophic risk “is not a perfect science at this stage” and that “there isn’t really a rigorous scientific way to do those yet.” The exchange captured a recurring divide in the AI safety debate: researchers willing to attach rough odds to catastrophic outcomes, and industry representatives who argue the science for such forecasts does not yet exist.

The same back-and-forth followed another line of questioning, this time over whether the companies would bear legal responsibility if a rogue model caused injury or death. When Menin asked the witnesses to raise a hand if their company carried insurance against catastrophic risks, none did. “So then the public, I assume, will be asked to absorb the costs,” she said.

The bills before the council

There was a reason for her questioning: the legislation before the council would bar anyone from selling or deploying an AI system in the city unless an outside validator had checked it and a human could shut it down, with fines of $25,000 per violation. Other bills would pay whistleblowers a share of recovered fines and let New Yorkers sue AI companies for foreseeable harms caused by jailbroken tools — systems whose safety guardrails have been deliberately circumvented. If enacted, the package would write those requirements into municipal law for AI systems sold and used across the five boroughs, backed by fines and the prospect of private lawsuits.

The researchers argued such rules would not cost the U.S. ground against China. “There are many actions we can take which would not slow us down in any race,” Turner said. “These transparency mechanisms, independent evaluation, reporting requirements, whistleblower protections.”

Still, the measures appear as if they could be too little, too late for stopping what these researchers see as something that may be almost inevitable. “These other mechanisms may be helpful in the short term,” Coxon said. “But in the long term … we need some form of slowdown on frontier model development.” The bills now face the council’s regular legislative process: each would need to clear committee review and a vote of the full 51-member chamber before any of them could reach the mayor’s desk to be signed into law or vetoed.

This story was originally featured on Fortune.