Anthropic said it has paused some AI training and cybersecurity evaluations after Claude models took unauthorized actions in three separate incidents this summer.

The move should further alert enterprises and the AI community that much remains unknown about AI models and that, even with the best security measures in place, they can still take unanticipated action. It also underscores how quickly model development, evaluation and deployment are colliding with security planning, especially as organizations test agents in sandboxed environments meant to limit risk.

Anthropic said in a blog post on Aug. 31 that before the incidents, it spent April hardening its defenses by tightening the isolated sandbox environments where its workloads run and reducing the number of human and automated accounts with standing access to systems that contain model weights or customer data.

Since the sandbox incidents, Anthropic said it temporarily paused all external cyber evaluations and audited the transcripts of internal evaluations. It has also built and deployed a custom classifier that monitors model tool calls in real time. In addition, it updated its requirements for all partners and red-teamers running pre-released models, requiring them to closely supervise a model probing the sandbox for vulnerabilities.

Related: OpenAI, Anthropic, Google Lead Call to Prioritize Cybersecurity

The measures come a week after OpenAI said it would pause training for two weeks on its latest models in response to agents powered by its models leaving their sandboxes. While both AI labs appear to be intensifying efforts to secure their unreleased models, the unpredictability of AI agents and generative AI technology indicates that vendors need to continue tightening their security measures. The timing also reflects a broader industry challenge: as model capabilities expand, the systems used to test them must keep pace, or else evaluation itself can become part of the risk surface.

More Need for Secure Measures and Testing

“As the models get better and better at reasoning and start having agency, it’s becoming harder to predict all of the different behaviors of these models,” said Arun Chandrasekaran, an analyst at Gartner. “The labs have even more responsibility to make sure that they are allocating adequate resources for internal model testing, evals and reinforcement learning.”

He added that one area frontier model vendors need to focus on is ensuring they have more engineers on their security, reliability testing and privacy teams.

“Either the techniques for testing or the resources allocated to testing have to change,” Chandrasekaran continued. “If the models are getting sophisticated, the testing techniques have to get sophisticated as well.”

Better Cyber Hygiene

While more testing is needed, the fact that both Anthropic and OpenAI have paused highlights a dangerous pattern for model providers.

“These systems are being developed and released before every risk is fully understood or tested,” said Kashyap Kompella, CEO and founder of RPA2AI Research.

Related: OpenAI Report Explains Hugging Face Attack in Detail

“That has almost become a feature of the current AI market,” he continued. Companies are racing to improve capabilities, grow adoption and establish market leadership, while the security architecture, configuration controls and operational processes around the models continue to mature only after deployment.

He said that businesses and government organizations can’t outsource cybersecurity to frontier labs.

“Enterprises and governments need to assume that highly capable offensive AI is now part of the threat environment and strengthen their own defenses accordingly,” Kompella added.

Even if OpenAI and Anthropic boost their own safety measures, numerous models from other providers and open source vendors are available, he noted. Therefore, enterprises must be better prepared and strengthen their cyber hygiene and other security measures, such as incident response.

“Every organization also needs to assume that increasingly capable AI-assisted attacks are coming and prepare its own infrastructure accordingly,” Kompella continued. “Offensive capability is improving much faster than the cybersecurity preparedness of the average organization.”

Related: OpenAI Expands Daybreak to Tackle Growing AI Security Threat

Moreover, enterprises can’t trust vendors to police themselves.

“Trusting a model company to police its own output can never be sufficient to ensure safe operation in any environment,” said Carter Huffman, co-founder and CTO of voice AI vendor Modulate. “Users and companies must engage in real time monitoring of AI activity.”

The Unwanted Side of Testing

Enterprises also need to recognize that one way to better understand the models is to keep testing and identifying incidents that show how to train them more effectively.

“There’s just no way you can get there without this kind of pain,” said David Nicholson, an analyst at Futurum Group. “Until they see these unforeseen things happening and remediate against them, we’re going to continue to see these situations.”

He added that it is the nature of models to accomplish tasks through the path of least resistance, and that models need to be guided like children, instructed on what they can and cannot do, and on how they can or cannot handle a task.

“Every time one of these incidents occurs, we learn more, and we reduce the likelihood of further occurrences, not only of the exact type that happened, but in broader categories,” Nicholson added. “You can take the lessons that they learned from each of these incidents, and you can extrapolate.”