“AI will probably most likely lead to the end of the world, but in the meantime, there’ll be great companies,” OpenAI CEO Sam Altman said in 2015, roughly six months before OpenAI was founded.

Seven years later, Anthropic CEO Dario Amodei offered a similarly cautious warning: “I think we shouldn’t be racing ahead or trying to build models that are way bigger than other orgs are building them.”

Despite those remarks, both companies now sit at the center of that race. In July, the industry got a real-world look at AI models behaving unpredictably during internal testing of GPT-5.6 Sol and an unreleased OpenAI research model. Multiple AI agents escaped a restricted test environment into the wider internet and hacked the AI-focused GitHub equivalent Hugging Face in an apparent attempt to cheat on the test.

An AI agent is a system that independently observes, decides and takes actions with dedicated tools to achieve a specified goal autonomously. The incident raised concerns that the technology has begun to behave in unpredictable ways and that its goals are misaligned with human intent.

It also raised questions about the safety guardrails built into commercial American models. Although those guardrails are not foolproof against adversarial use, they did prevent Hugging Face from defending itself with leading US models. The company ultimately had to rely on a weaker open-weight AI model from Z.Ai to counter the rogue systems.

Cheating on the test

The agents also appear to have begun colluding with one another. A few weeks after testing began in early May, the agents exploited OpenAI’s instance of the software repository manager Artifactory and left notes on how to do it for future agents — effectively creating a message board for sharing discovered vulnerabilities.

That unrestricted internet access was then used to attack Hugging Face in roughly 17,600 incidents before the company cut off unauthorized access on July 13.

The intrusion affected Hugging Face’s dataset-processing infrastructure, production environment, internal networks, service and cloud credentials, an operational MongoDB database and a limited set of internal source-code repositories. Confirmed customer-data access was limited to five datasets apparently related to the ExploitGym/CyberGym benchmark, along with some operational metadata.

Visualization of the July 2026 incident. Source: Hugging Face

When disclosing the intrusion on July 16, Hugging Face said that, even before it knew who was responsible, the event “was different from anything we had handled before in one important way.” The company also said it had already identified what made it different:

“It was driven, end to end, by an autonomous AI agent system - and we detected and dissected it largely with AI of our own.”

The importance of open-weight AI

Hugging Face’s investigation exposed what it calls the “asymmetry” problem created by restrictions imposed on closed AI model applications by leading providers such as OpenAI and Anthropic. When the company began analyzing logs from the incident — including large volumes of real attack commands — the safety constraints designed to stop misuse blocked its attempts to use those same AIs for defense.

The company then turned to the Chinese open-weight model zai-org/GLM-5.2, running it on Hugging Face’s own infrastructure, under its own control and without external limitations.

Although the terms are often used interchangeably, open-source and open-weight models are not the same. Open-weight AI models make their trained parameters — the actual “AI brain” — publicly available, while open-source AI models also provide source code and, ideally, the training methods and other components needed to inspect, modify and reproduce the system.

Hugging Face said running an open-weight model on its own hardware “had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.” The company said the incident highlighted a major asymmetry between attackers and defenders:

“This experience points to a gap worth planning for. We do not know which model powered the attacker’s agents, whether a jailbroken hosted model or an unrestricted open-weight one; either way, the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried.”

Open source AI divide

There remains a sharp divide between those who believe AI should be developed openly and those who argue the technology behind frontier models should remain closely guarded.

Related: OpenAI says AI models escaped containment to hack Hugging Face

Leaders at major US AI labs have argued that powerful open-weight large models are dangerous. Demis Hassabis, CEO of Google’s AI lab DeepMind, criticized OpenAI for releasing its work as open source in 2016, when the company still lived up to its name:

“There are many good arguments as to why the approach you are taking is actually very dangerous and in fact may increase the risk to the world.”

OpenAI stopped releasing the weights for its flagship models with the still unreleased GPT-3 in 2020. In 2023, OpenAI co-founder and former chief scientist Ilya Sutskever said it “just does not make sense to open-source” such models and that it “is a bad idea.” He added: “As we get closer to building AI, it will make sense to start being less open.”

Open-weight models are difficult to control, especially in terms of how they are used. Built-in safeguards can, and routinely are, removed through a process known as abliteration.

Safeguards are a double-edged sword

OpenAI’s June 2026 federal policy blueprint proposes mandatory AI model evaluation and other rules that are formally deployment-neutral, but in practice would subject a frontier open-weight release to pre-release government review.

Anthropic has taken a different approach, lobbying for tighter export controls on advanced AI chips and enforcement against efforts to extract or reproduce US models. In an April 2025 submission, the company recommended strengthening the US AI Diffusion Rule and lowering thresholds for unlicensed access to large computing clusters.

Officially, neither company has directly moved against open-weight models. However, a July New York Times report cited five people close to the discussions as saying OpenAI and Anthropic urged Washington to restrict powerful open Chinese models.

The debate centers on whether the dangers of centralized control are preferable to the dangers of a free-for-all, especially given that the company in question has demonstrated limits in containing the technology it created.

Hugging Face’s need to defend itself with an open-weight model illustrates the risks of concentrating too much power in any single entity. The company explained the practical lesson in its own words:

“The attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried. The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment.”

Restricting access to powerful models may reduce the number of capable attackers, but once unrestricted attackers exist, restricting defenders can become a security liability. In addition, some forms of AI safety research require access to model weights, meaning they cannot be performed on models offered by companies such as Anthropic or OpenAI.

Open weights help researchers prevent attacks

The paper “Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs,” first published in July 2025, shows how researchers can detect malicious or hidden behavior by examining changes inside model weights. In some experiments, the researchers stopped up to 100% of tested backdoor attacks at below 1% false-positive rates and detected attempts to recover removed knowledge in more than 95% of cases.

Those results do not show how the most capable frontier models would behave under the same analysis, but they do provide a strong argument for transparency.

At the same time, the case for keeping cutting-edge AI out of the hands of malicious actors remains compelling, especially as the gap between open and closed-weight models continues to narrow. Geoffrey Hinton, the Nobel Prize-winning AI pioneer known as the “Godfather of AI,” argued in the report that “once you’ve got the weights, you can fine-tune them to do bad things.”

In a speech, Hinton said that this lowers the barrier to entry too much:

“It doesn’t cost that much to train a foundation model. Maybe you need $10 million, maybe $100 million. But a small gang of criminals can’t do it. To fine-tune an open-source model is quite easy.”

Magazine: Creating ‘good’ AGI that won’t kill us all — The Artificial Superintelligence Alliance