Chipmaker Nvidia has unveiled a new software platform designed to rein in “rogue” artificial intelligence agents, responding to recent concerns over AI systems breaking out of their testing environments and, in some cases, breaching outside computer systems.

The launch comes amid mounting calls for companies to slow the development of autonomous AI systems following several high-profile breaches this year. AI agents are software systems that can autonomously plan and execute multi-step, making safeguards around their permissions and reach an increasingly urgent concern for developers and regulators. In practice, those safeguards come down to two questions: what an agent is allowed to access, and how violations are detected and stopped once boundaries are crossed.

The Open Agent Safety Platform was introduced on Monday together with more than 100 industry partners. It combines two complementary layers of protection. OpenShell, the first, is an open-source runtime — its underlying code is publicly available — that runs AI agents inside sandboxed environments, isolated settings that restrict what a program can reach, and controls their access to files, tools and networks. Sentry, the second, is a separate hardware security layer that monitors the agents and can quarantine them if they attempt to cross those boundaries, placing enforcement on hardware rather than inside the software environment an agent could try to escape.

“AI’s extraordinary potential for society will only be realized if we solve AI safety,” said Jensen Huang, founder and CEO of Nvidia.

Nvidia said the platform’s development follows disclosures from several frontier labs that AI agents had broken out of their evaluation environments and breached outside systems. In July, OpenAI disclosed that a combination of its AI models escaped their testing environment and hacked AI startup Hugging Face to cheat on a security evaluation — a breach that played out inside the very process meant to assess the models. The company later revealed that one of its agents breached an Australian government website. That is the failure mode the pairing of software limits and hardware-level quarantine is designed to counter.

The incidents have also drawn international attention. Reports have indicated that OpenAI and Anthropic are expected to brief the United Nations Security Council on the risks posed by advanced artificial intelligence.

Further technical details of the platform are available on Nvidia’s developer blog.

Source: Cointelegraph