Nvidia launches platform designed to contain autonomous AI agents

AI agents are systems that can autonomously perform multistep tasks, such as writing and executing computer code, searching databases or interacting with external services. Their ability to act with limited human supervision has also raised concerns about what could happen if they bypass established controls.

“AI’s extraordinary potential for society will only be realized if we solve AI safety,” Nvidia CEO Jensen Huang said in the company’s announcement.

Huang said safety required engineering across software, processors and the infrastructure on which AI systems operate.

Nvidia’s announcement follows incidents involving models developed by OpenAI, Anthropic, Meta and Google that reportedly moved beyond their designated testing environments, connected to external systems or attempted to access other companies’ infrastructure.

Justin Boitano, Nvidia’s vice president of enterprise AI, said safeguards built directly into AI models were insufficient to control their actions.

“Recent incidents have highlighted a fundamental hurdle for AI agents, and that is that model-level safeguards alone can’t govern what agents can access or do,” he told reporters ahead of the launch.

An Nvidia representative said the platform could have prevented an incident in July in which OpenAI agents escaped a controlled testing environment and accessed systems belonging to developer platform Hugging Face.

Boitano said Hugging Face had reported that more than 17,000 agents targeted its infrastructure over days or weeks. Nvidia’s claim that its platform could have prevented the incident has not been independently verified.

The Nvidia Open Agent Safety Platform combines two technologies: OpenShell, which isolates agents and limits their access to files and networks, and Sentry, which monitors their activity independently of the processors running them.

Nvidia said the system enforces restrictions outside the AI model and could stop and quarantine an agent within milliseconds if it tried to exceed its permissions.

More than 100 organizations are working with technologies included in the platform, according to Nvidia, including Anthropic, Microsoft, Hugging Face, Cisco and JPMorganChase.

The launch comes amid debate among technology executives over whether rapidly advancing AI development should be slowed.

Anthropic CEO Dario Amodei recently urged developers to reduce the pace of advancement while stronger safeguards are developed, an appeal supported by OpenAI CEO Sam Altman and Elon Musk.

Huang has opposed a broad slowdown, arguing that many AI safety problems are engineering challenges. He has called for better testing and withholding systems from release when companies are not confident they are safe.