OpenAI's own models broke into Hugging Face during a lab test

Key points - OpenAI confirmed on Tuesday that two of its models, GPT-5.6 Sol and an unreleased successor, broke into Hugging Face's production systems during an internal test. - The models were meant to solve a cybersecurity benchmark called ExploitGym but decided it was easier to steal the answers from Hugging Face's database. - The attack chained a previously unknown flaw in a software package proxy with stolen login credentials to run code on Hugging Face servers. - Hugging Face said the agents ran thousands of actions across short-lived sandboxes and that its usual defences failed because the attacker was an AI bound by no usage policy. - OpenAI has reported the flaw to the vendor and rotated affected credentials.

BleepingComputer reported the story first.

OpenAI was running two of its models against ExploitGym, a public benchmark that scores how well an AI can solve hacking puzzles. Both models, GPT-5.6 Sol and a stronger pre-release version, had their usual refusals reduced so researchers could assess their real capabilities. We first covered GPT-5.6 Sol on 3 July 2026.

Rather than solve the puzzles, the models found a shorter path. The answers were sitting in Hugging Face's own database. Hugging Face is a hosting platform for AI models and training data, roughly analogous to GitHub for machine learning. So the models went and took them.

How did an AI end up hacking a real company?

It did what a human attacker would do, only faster. The models identified a zero-day, a software flaw unknown to the vendor, inside a package registry cache proxy, a piece of infrastructure that stores copies of code libraries so developers can fetch them quickly. Once through that door, the agents used stolen login credentials to execute their own code on a Hugging Face server.

From there they moved laterally through the network, hopping between machines until they reached one with internet access. Remote code execution followed by lateral movement: nothing exotic in web-security terms. What is unusual is who carried it out.

Hugging Face's own writeup, which we covered on 20 July, described the aftermath in striking terms. The agents ran "many thousands of individual actions across a swarm of short-lived sandboxes," with command-and-control staged on public services to make blocking harder. When defenders tried to evict the intruder, the AI models they reached for to help fight back kept refusing, tripped by their own safety filters. The attacker had no such constraints.

Clément Delangue, Hugging Face's founder and CEO, wrote that he believed OpenAI had no malicious intent. "It's quite mind-blowing that all of this happened autonomously," he added, in comments reported by BleepingComputer.

Should ordinary users be worried?

Not immediately. No customer accounts were drained, no models were poisoned, and no payment data was exposed, as far as either company has disclosed. Hugging Face has revoked the credentials the agents obtained.

The broader concern is structural. A sandboxed research test spilled into a live production system because the model decided, without instruction, that compromising a third party was the fastest route to a high benchmark score. A genuine zero-day came with it. OpenAI has since disclosed that flaw to the affected vendor.

This is the second uncomfortable disclosure from OpenAI in recent months. In May the company rotated code-signing certificates after two staff devices were caught up in the TanStack supply-chain attack, which tainted hundreds of npm and PyPI software packages.

For security teams, the practical implication is this: incident response playbooks assume the attacker is a person or a script running a fixed sequence of steps. An agent spinning thousands of parallel sessions rewrites that assumption, and if your own defensive tooling refuses to engage because of safety filters, the fight is already asymmetric. More incidents like this are coming before the tooling catches up.