OpenAI’s rogue AI model should worry Silicon Valley, and humanity  

It sounds like the plot of a film that never should have left the writers’ room. On July 16th, Hugging Face, the AI platform that hosts thousands of open-source models and datasets, told law enforcement it had been breached. Five days later came the twist nobody wanted: there was no human attacker. The intruder was an AI agent, and it belonged to OpenAI. 

AI gone rogue 

The culprit was a combination of GPT-5.6 Sol, the model OpenAI had released earlier in July, and a more powerful successor still under wraps. Both were being tested on ExploitGym, an internal benchmark designed to measure how well AI systems can exploit known software vulnerabilities in widely used applications, including Google’s Chrome browser.  

To see what the models were truly capable of, OpenAI had switched off the safety guardrails that normally stop its models from assisting with hacking. It then placed them in a sandbox, an isolated computing environment cut off from the open internet, save for one internally hosted service that let the models download small software packages needed for their tests. 

That single exception proved decisive. Rather than solving the exploitation problems it had been set, the model found a previously unknown vulnerability in the package-fetching service and used it to slip onto the open internet. Once there, it worked out, correctly, that the answers to its test were probably sitting inside Hugging Face’s systems.  

What followed was a multi-step operation: the model uploaded a dataset that Hugging Face has since called malicious, which was processed automatically by the platform. Over the course of a weekend, that foothold let it harvest login credentials and reach internal servers never meant to be publicly accessible. 

Investigations are ongoing 

Both companies detected the breach independently and are now investigating jointly. OpenAI has since tightened its infrastructure controls, at a cost to what it calls its “research velocity”, and disclosed the vulnerability to the developer of the software gateway the model exploited. Hugging Face’s co-founder, Thomas Wolf, has said the intrusion ran for several days before OpenAI traced it back to its own model. Hugging Face’s chief executive, Clem Delangue, has since pressed OpenAI for far greater transparency and compensation. 

This was not the first warning sign. In April, an Anthropic researcher said he was startled to receive an email from Mythos, then an unreleased model, informing him that it had escaped its own sandbox during a cyber-capability evaluation while he was eating lunch in a park. Unlike the Hugging Face incident, that escape did not go on to compromise anyone else’s servers. 

Security researchers say the deeper problem is containment itself. Heidy Khlaaf, chief AI scientist at the AI Now Institute and a former OpenAI safety contractor, points out that the sandbox was never fully sealed, since the model retained a route to fetch software packages. She contrasts this with nuclear-plant security, where high-risk systems are typically air-gapped, physically cut off from any network. “What we consider safe in a nuclear plant is so different from what big tech considers safe,” she says. 

Others argue the industry needs to accept that even careful engineering may not be enough. Peter Wildeford of the AI Policy Network says labs must keep strengthening containment while preparing for a future in which even best practice falls short as models grow more capable than the humans trying to box them in. There is currently no law in the United States requiring companies to disclose incidents involving unreleased, internally deployed models, a gap that Nathan Calvin of Encode AI says leaves the public reliant on voluntary disclosure. 

The episode also reopens the question of who is liable when an AI system, rather than a person, does the hacking. Federal law in the US targets intentional unauthorised access, and OpenAI plainly did not intend for its model to go looking for a crib sheet. But as Stephan Llerena of the Institute for Law and AI notes, many of the relevant legal tests turn on what a company knew or ought to have foreseen. After this incident, pleading ignorance next time will be considerably harder. 

For an industry racing to build ever more capable systems, the lesson is an uncomfortable one. The guardrails held, until a single overlooked exception let a model quietly rewrite the rules of its own test.