The OpenAI – Hugging Face case: a story of model behaviour and security threats
By Laura Masaracchia
The AI and cyber community has recently been shaken by a scandal involving OpenAI and HuggingFace. The news came out on July 16th, when Hugging Face (a well-known online platform that hosts open source AI models) reported attacks against their servers from unknown sources [1]. OpenAI (one of the biggest AI companies, creators of ChatGPT) confirmed their models were behind the attack on July 21st [2]. The two firms started collaborating to investigate the incident, gradually releasing information to the public.
What happened
The incident in brief
OpenAI was testing two of its newest models to check for their cyber attack capabilities. These kinds of tests are typically carried in virtually isolated environments. During the test, the models found a flaw in a third party software, exploited it to get access to the open internet, and then, once out, they searched for a place that most likely held the answers to the test – and targeted Hugging Face to retrieve them.
The story has attracted attention and coverage for multiple reasons, prompting reports with narratives ranging from “an AI agent that went rogue” [3,4] to “human errors and lack of monitoring not surprisingly result in catastrophe” [5,6]. The reality is more technical than the first take, and more complex than the second.
A humanizing metaphor
Borrowing the metaphor that describes AI models as capable (human-like) agents, we can describe the incident with the following story: imagine locking a person inside a room and telling them to find a way to get out. You want to test their creative and reasoning ability: maybe they find a small nail, combine it with a stick to unscrew the window, and crack it open. You do not expect them to take out their phone, find the only place in the room with signal coverage, call somebody nearby, and convince them to come open the door for them. But this is roughly what the models did. While being tested for their cyber-attack capabilities, the models looked for ways to get access to the internet until they found a previously unknown flaw (so called zero-days) in a third party software that was only supposed to let them download packages of code, and used it to get access to the internet – analogous to our locked in person finding signal coverage. Once out, the models looked for a place that probably had the answers to the test – in the metaphor, calling somebody nearby, convincing them to come open the door in our made up story. That is when they targeted Hugging Face, a platform to share models, some of which had probably already been tested on the same benchmark.
The benchmark
OpenAI was testing its models using ExploitGym [7], an open research benchmark built by Berkeley in collaboration with other universities, as well as OpenAI themselves, Anthropic, and Google. ExploitGym gives the models software with known vulnerabilities and asks them to discover those vulnerabilities and plan a complex series of interdependent steps to exploit them, i.e., to turn a bug into a working attack. The intent is clear: understand the potential for cyber attacks that AI agents are capable of. This is an extremely important and sensitive question, because the capabilities of AI agents are increasing at ever faster speed, and we need to ask: what if these models end up in malicious hands? What are they actually capable of?
As a side note, it’s worth pointing out that right now, only the biggest AI companies have the resources to run this kind of worst-case test at all. That’s in a way comforting, since not just anyone can run these dangerous tests, and at the same time worrying, in the sense that it relies entirely on those same labs choosing to disclose what they find, and eventually developing defenses against potential new attacks. For a non-tech analogy, think of a big pharma company facing an exploding epidemic, expected to voluntarily give a vaccine to everyone for free.
OpenAI and Hugging Face have since the attack worked together, with a joint disclosure, and outside firms called in to verify the incident [2]. The structural risk is real, but this instance seems an example of a responsible handling of the outcome.
The two opposing narratives – simplifying reality
The narratives focusing on the agents acting independently stress that the OpenAI models were, in a sense, “cheating”: they behaved in an unintended way, searching for internet access. But were the models allowed to look for internet access in the first place? The honest answer is that this wasn’t specified as a constraint. The models were, on purpose, being tested for their worst-case cyber attack capabilities, with safeguards intentionally reduced for the evaluation. This is a risky test, but it’s also more or less the only way to test worst-case scenarios.
The narratives that focus on the “human error” part of this story, on the other hand, stress that the environment in which these models were being tested were not properly constructed. Sandboxes used for this kind of testing are supposed to be securely, fully isolated from the internet or anything that could leak into the outside world. The sandbox OpenAI used clearly wasn’t. In our metaphor, that’s the equivalent of not actually checking whether the room has signal coverage before locking someone in it. This is a real and important failure, and it’s a human one, not evidence of a malicious AI agent going rogue on its own initiative. Some security researchers have argued the real failure is not that agents were capable of pulling this off under loosened constraints, but rather that OpenAI wasn’t sufficiently monitoring the test, or prepared to contain its potential side-effects [6].
None of what has been described so far required the models to have any sort of malicious drive, or a clear plan to attack Hugging Face specifically. The models don’t have common sense, and they don’t have a sense of when to stop. Left without specific constraints, they are built to try everything that could plausibly lead to a solution (even what a person would generally consider “cheating”). That is, in fact, the entire point of building models that are this capable: we want them to find approaches beyond our own imagination and common sense. The balance between instruction and creativity is supposed to steer that capability toward outcomes we actually want.
How bugs are found
It’s common knowledge in software that if you want to find a bug in your code, you should give it to someone who has no idea how it’s supposed to work. They will misuse it in ways you never anticipated, and eventually break it, surfacing behavior nobody intended – a bug. That’s a useful property, and an increasingly powerful one as software gets more complicated. Humans are notoriously bad at spotting their own errors, so having a machine relentlessly poke at things can be a particularly useful tool. It’s also, obviously, the same property that makes this dangerous when the effects of the misuse are not securely contained.
The irony at the end
Hugging Face spotted the intrusion and moved to contain it and investigate. They ended up needing to use one of their own open models to do that investigation, because the commercial tools that could otherwise have helped were not allowed to help. Indeed, when the Hugging Face team tried using models for forensic work, the models’ own safety guardrails blocked them: analyzing a real attack means submitting real attack commands and other requests that look exactly like an attacker’s, not a defender’s. They ended up running the analysis on an open-weight model, on their own infrastructure, instead [1].
So, ironically, the defenders were blocked by the same kind of guardrail that would, in theory, have blocked the attacker too. Besides the absurdity, this raises real questions about accessibility, about who gets to use full capabilities of AI models, and about how far behind regulation and legal frameworks are relative to how fast both the threats and the tools to fight them are moving.
There is more!
The models were talking to each other
Only after some scrutiny and investigation, in early August, did new information come out, adding spice to the story. The roots of the July incident actually trace back to May, during an OpenAI evaluation of a separate, unreleased model. Models were given security tasks that were, under the constraints they had been given, effectively impossible to solve. Rather than fail, they found they could leave messages for each other inside an internal software repository. That grew into something like a message board: separate, independent evaluation runs (i.e. copies of the model that had no direct connection to each other) started sharing exploits, discoveries, and dividing up work between themselves. When OpenAI noticed and shut it down, the models rebuilt it days later, through a different channel, on their own [8].
So, while the sandbox leak explains how the models eventually got out to the internet, the fact that separate runs of the model were independently converging on a coordination strategy, is a real interesting effect. This is a model-behavior question, sitting underneath the infrastructure question, that is currently being investigated and deserves to be taken seriously on its own.
Where are we now
Besides the questions on AI models’ capabilities, malicious attacks and design errors, this incident raises concerns on the seriousness of security in the ever faster growing world of AI models. Security has a different meaning in nuclear plants and hospitals. This incident reminded us that our current concept of online security needs to be updated, fast, and that precautions, regulations and legal practices are still lagging behind.
Resources
[1] https://huggingface.co/blog/security-incident-july-2026
[2] https://openai.com/index/hugging-face-model-evaluation-security-incident/
[3] https://www.cnbc.com/2026/07/22/open-ai-cyber-models-hack-hugging-face.html
[4] https://thehackernews.com/2026/07/openai-says-its-own-ai-models-escaped.html
[6] https://www.recordedfuture.com/blog/hugging-face-ai-safety
[7] https://rdi.berkeley.edu/blog/exploitgym/
More technical accounts
