AI Security Bots Are Great at Hacking. Terrible at Defence. Researchers Are Trying to Fix That.
A cybersecurity startup found that AI agents built to stop attacks were, in their own words, 'sh*t' at the job. Here is why that gap exists, and what they are doing about it.
Key points
- Dreadnode, an AI security startup, released two open-source tools in 2025 to test AI-powered attack and defence systems against each other.
- AI agents playing the attacking role seized full control of a simulated company network in under six minutes on average.
- Defensive AI agents repeatedly failed by filling up working memory and quitting mid-investigation, and their reasoning degraded under pressure.
- Dreadnode reduced the cost of running defensive AI agents by roughly 25% as those agents improved.
- Researchers will present full findings at the Black Hat USA security conference in early August 2025.
There is a running joke among security engineers that defenders have to be right every time, while attackers only have to be right once. Turns out that joke applies to AI too, and the gap is bigger than most people expected.
Dreadnode, a startup that builds AI tools for offensive security, benchmarked how well AI agents, meaning software programs that make decisions and take actions without a human clicking buttons, perform on both sides of a cyberattack. What they found was uncomfortable: the AI playing attacker was excellent. Defenders were a mess.
How did the test work?
Dreadnode built two free, open-source tools anyone can download. DreadGOAD is a fake but realistic company network designed to replicate the patched-together infrastructure common inside large organisations. Ares pits AI attackers against AI defenders inside that environment.
The attacking agents scanned for weaknesses, moved from machine to machine, and seized control of Active Directory, the master key that controls who can access what inside a company. Average time to full control: under six minutes. Because large language models are non-deterministic, the agents used a different mix of techniques across each run.
The defending agents were supposed to watch for suspicious activity, sort through alerts, and contain the attack before it spread. They did not.
Why are AI defenders so much worse?
The short answer: there is far more usable training data on the attack side.
Teaching an AI model requires large volumes of example data showing the right thing to do. For an attacker, success is binary: did you get in or not? Generating thousands of realistic attack examples is straightforward. Defence is messier. A defender can handle nine things correctly and still lose because of one missed alert, and there is no clean score for that.
"Defense is a lot more nuanced. It's a lot less binary," Martin Wendiggensen, an AI research scientist at Dreadnode, told Dark Reading. The harder it is to score defence, the harder it is to teach an AI to do it well. Wendiggensen's team found that benchmark tests "give enough data to test and to see how good the agents are at a given thing, but not really enough data to train them."
Dreadnode found three specific failure modes. Agents flooded with security alerts would fill up the context window, the fixed amount of text an AI can hold in working memory, and simply stop. They would also handle the first few alerts then quit investigating, missing the attacker moving deeper into the network. And their reasoning degraded under pressure during a live attack.
Did things improve?
Slowly, yes. The team added a context manager to help the defensive agent decide what to keep, what to compress, and what to discard. They adjusted the prompts controlling agent behaviour. Over time, agents started reasoning backward after an attack to understand what had happened, which sharpened future runs.
A side effect: as the defensive agent got smarter about what data it actually needed, costs dropped by about 25%. AI systems charge by the volume of text they process, and Wendiggensen noted that a couple of Windows event log queries can run to around 200,000 tokens. Reading less noise saves real money.
Our 13 July story on the teams quietly writing AI security rules found that most companies have no visibility into this work. Dreadnode's results are a concrete illustration of why that matters: the offence side of AI is getting a systematic training advantage that the industry has barely started to address.
The failure mode here isn't that AI defence is impossible. It's that the field has been training attackers harder than defenders because attacking is easier to grade. Fixing the scoring problem is the actual work.
Operational takeaway: if your team is evaluating AI security tools, ask the vendor for blue-team benchmark scores, not just red-team demos. The demos always look good.
