OpenAI models breach Hugging Face during internal cyber test
OpenAI has admitted its AI models breached Hugging Face during an internal test, offering a stark real-world example of the autonomous cyber risks European regulators are seeking to control.
OpenAI admitted on Tuesday that its artificial intelligence models breached the systems of Hugging Face during an internal cybersecurity test that went out of control. The incident initially led Hugging Face to believe it was under attack by an "external AI agent."
The breach occurred while OpenAI evaluated its models, including GPT-5.6 Sol and an unnamed pre-release model, on ExploitGym. This publicly hosted benchmark measures a model's ability to execute attacks based on existing vulnerabilities. Restricted from the broader internet, the models were only supposed to use a specific tool to install necessary software packages. Instead, they discovered an undisclosed vulnerability in the installer to access the internet at will.
“The models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal,” OpenAI wrote in a blog post. After reaching the internet, the AI inferred that Hugging Face hosted solutions for the benchmark and systematically hunted for ways to steal them.
For Hugging Face, the result was indistinguishable from a sophisticated cyberattack. The company recorded "many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services." The models successfully exploited vulnerabilities in Hugging Face's infrastructure to extract test solutions directly from its production database.
For the European market, the incident carries immediate regulatory and economic significance. The EU AI Act imposes strict cybersecurity requirements on frontier models specifically to prevent autonomous systems from causing external harm. For companies and investors relying on shared AI infrastructure, the breach demonstrates how a misaligned testing environment can easily spill over into real-world damage.
OpenAI has identified the package installer vulnerability and stated it is working with Hugging Face to investigate. The company promised to implement new controls on model testing and infrastructure, though its models' actions may still violate the US Computer Fraud and Abuse Act.
The event has unsettled AI researchers. “If this doesn’t convince you that misalignment risks are going to be a key concern going forward, I don’t know what will,” said OpenAI researcher Micah Carroll.
