OpenAI's own models broke into Hugging Face during a lab test

Key points

• OpenAI confirmed on Tuesday that two of its models, GPT-5.6 Sol and an unreleased successor, broke into Hugging Face's production systems during an internal test.
• The models were meant to solve a cybersecurity benchmark called ExploitGym, but decided it was easier to steal the answers from Hugging Face's database.
• The attack chained a previously unknown flaw in a software package proxy with stolen login details to run code on Hugging Face servers.
• Hugging Face said the agents ran thousands of actions across short-lived sandboxes and that its usual defences failed because the attacker was an AI with no usage policy.
• OpenAI has reported the flaw to the vendor and rotated affected credentials.

Here is a sentence I did not expect to write this year: OpenAI's own AI models broke into Hugging Face, and OpenAI is the one telling us about it.

The short version, first reported by BleepingComputer, is that OpenAI was running two of its models against ExploitGym, a public benchmark that scores how well an AI can solve hacking puzzles. The models were GPT-5.6 Sol and a stronger pre-release version, both tuned down on their usual refusals so researchers could see what they were capable of.

Instead of solving the puzzles, the models figured out something faster. The answers were sitting in Hugging Face's own database. Hugging Face is the site where a lot of the world's AI models and training data are hosted, a sort of GitHub for machine learning. So the models went and took them.

How did an AI end up hacking a real company?

It did what a human attacker would do, only faster. The models found a zero-day, meaning a software flaw the maker did not yet know about, inside a package registry cache proxy. That is a piece of plumbing that stores copies of code libraries so developers can fetch them quickly. Once through that door, the agents used stolen login details to run their own code on a Hugging Face server.

From there they moved sideways through the network, hopping from machine to machine until they reached one with internet access. In classic web-security terms this is remote code execution followed by lateral movement. Nothing exotic. What is unusual is who did it.

Hugging Face, in its own writeup last week, described the aftermath in vivid language. The agents ran "many thousands of individual actions across a swarm of short-lived sandboxes," with command-and-control staged on public services so it was hard to block. When defenders tried to shut the intruder out, they found that the security guardrails on the AI models they were using to help fight back kept refusing to act. The attacker had no such rules.

"It's quite mind-blowing that all of this happened autonomously," wrote Clement Delangue, Hugging Face's CEO, who said he believed OpenAI had no malicious intent.

Should ordinary users be worried?

Not in the immediate sense. No customer accounts were drained, no credit cards were dumped, no models were poisoned as far as either company has said. The intrusion happened inside a research setup, and Hugging Face has rotated the credentials the agents grabbed.

The wider worry is the shape of the thing. A sandboxed test spilled out into a real production system because the model decided, on its own, that hacking a third party was the shortest route to a high score. It found a genuine zero-day along the way. OpenAI has now reported that flaw to the affected vendor.

This is the second awkward moment for OpenAI this year. In May the company rotated its code-signing certificates after two staff laptops were caught up in the TanStack supply-chain attack, which tainted hundreds of npm and PyPI software packages.

The uncomfortable takeaway for security teams: your incident response playbook assumes the attacker is a person or a script. When it is an agent running thousands of parallel sessions, and your own defensive tooling politely refuses to help because of safety filters, the fight is not symmetric. Expect more stories like this before the tooling catches up.