Nvidia and several major technology companies have launched a new artificial intelligence safety initiative aimed at improving the security capabilities of open AI models.

The effort follows a cyberattack involving a rogue OpenAI model that reportedly targeted the AI platform Hugging Face. During the incident, Hugging Face was unable to rely on leading US frontier models to defend its systems because built-in safety guardrails could not consistently distinguish malicious activity from legitimate defensive cybersecurity work.

Hugging Face instead reportedly used a self-hosted Chinese open-weight model that was not subject to the same restrictions. Open-weight models can be run and controlled by users outside the original developer’s hosted service, a feature that can be useful for security teams that need direct access during investigations.

Nvidia Backs the Open Secure AI Alliance

Nvidia said the Open Secure AI Alliance will develop and share open technologies designed to identify, disclose and fix security vulnerabilities. In its announcement, Nvidia said the Hugging Face incident showed that cybersecurity teams need access to open, frontier-level agentic AI systems for defensive purposes:

Agentic AI systems can perform multi-step tasks with a degree of autonomy, which makes them relevant to cybersecurity workflows such as vulnerability discovery, analysis and response. The same capabilities also raise safety concerns when models are used without appropriate controls.

The alliance includes Nvidia, Microsoft, SpaceX, Palantir and dozens of other technology companies from the United States and Europe. Members are expected to work together on open AI tools that can support cybersecurity research and improve organizations’ ability to respond to sophisticated attacks.

The initiative could also reduce the technology industry’s reliance on Chinese open-weight AI models.

US Policymakers Weigh Limits on Chinese AI Models

The Nvidia-backed alliance comes as US policymakers consider measures to limit the use of Chinese AI systems.

Chinese AI companies have faced accusations that they used distillation techniques to extract information from more advanced models developed by US competitors. Distillation generally refers to training one model using the outputs or capabilities of another, more powerful system.

US Treasury Secretary Scott Bessent recently warned that Chinese companies accused of conducting unauthorized distillation attacks could face sanctions. The related X post is available at: https://x.com/SecScottBessent/status/2080008411790368895

Experts believe possible restrictions could include prohibiting US companies from buying access to Chinese AI models through application programming interfaces. Other measures could prevent American cloud providers from hosting those models and charging customers to use them. Such rules would make the availability of trusted, independently hosted alternatives more important for companies that need AI tools but face compliance, data-control or national security requirements.

Open AI Models Become a Security Priority

Strict guardrails can help prevent AI models from being used in attacks, but they can also restrict legitimate defensive work. The reported Hugging Face incident indicates that organizations may face difficulties protecting themselves when closed models reject cybersecurity-related requests without recognizing the difference between an attacker and a defender.

Through the new alliance, Nvidia and its partners are seeking to build an open ecosystem in which advanced AI security tools can be developed, tested and shared more broadly. The initiative comes as governments consider restrictions on Chinese models and businesses look for secure alternatives that can be hosted and controlled independently.