AI Security Bots Are Great at Hacking. Terrible at Defence. Researchers Are Trying to Fix That.
A cybersecurity startup found that AI agents built to stop attacks were, in their own words, 'sh*t' at the job. Here is why that gap exists, and what they are doing about it.
Key points
- Dreadnode, an AI security startup, released two open-source tools in 2025 to test AI-powered attack and defence systems against each other.
- AI agents playing the attacking role took full control of a simulated company network in under six minutes on average.
- Defensive AI agents repeatedly failed by running out of memory, quitting mid-attack, and losing reasoning ability under pressure.
- Dreadnode reduced the cost of running defensive AI agents by roughly 25% as those agents got better at their jobs.
- Researchers will present full findings at the Black Hat USA security conference in early August 2025.
There is a running joke among security engineers that defenders have to be right every time, while attackers only have to be right once. Turns out that joke applies to AI too, and the gap is bigger than most people expected.
Dreadnode, a startup that builds AI tools for offensive security, set out to benchmark how well AI agents, meaning software programs that make decisions and take actions on their own without a human clicking buttons, perform on both sides of a cyberattack. What they found was uncomfortable: the AI playing attacker was excellent. The AI playing defender was a mess.
How did the test work?
The company built two free, open-source tools anyone can download. The first, called DreadGOAD, is a fake but realistic company computer network designed to look like the kind of messy, patched-together setup you actually find inside large organisations. The second, called Ares, pits AI attackers against AI defenders inside that fake network.
The attacking agents did what real criminals do: they scanned the network for weaknesses, moved from machine to machine, and tried to seize control of a system called Active Directory, which is essentially the master key that controls who can access what inside a company. They succeeded, on average, in under six minutes.
The defending agents were supposed to watch for suspicious activity, sort through security alerts, and shut the attack down before it spread. They did not do well.
Why are AI defenders so much worse?
The short answer: there is far more practice data available for attacking than for defending.
Teaching an AI model requires enormous amounts of example data showing the right thing to do. For an attacker, success is simple to measure: did you get in, yes or no? Generating thousands of realistic attack examples is straightforward. Defence is messier. A defender can do ten things right and still fail because of one missed alert. There is no clean, binary score.
"Defense is a lot more nuanced. It's a lot less binary," said Martin Wendiggensen, an AI research scientist at Dreadnode, as first reported by Dark Reading. The harder it is to score defence, the harder it is to teach an AI to do it well.
Dreadnode found three specific ways their defensive agents broke down. First, the agents ran out of what is called a "context window," which is the amount of information an AI can hold in its working memory at one time. Flooded with security alerts, the agent simply stopped. Second, the agent would handle the first few alerts and then quit investigating, missing the attacker moving deeper into the network. Third, its reasoning got worse, not better, as the attack intensified.
Did things improve?
Yes, slowly. The team added a memory manager to help the defensive agent decide what to keep, what to store, and what to discard. They adjusted how the agent was instructed to behave. Over time, the agent started working backwards after an attack to understand what had happened, which made it sharper in future tests.
A side effect: as the defensive agent got smarter about what data it actually needed to look at, the cost of running it dropped by about 25%. AI systems charge by the amount of text they process, so reading less junk data saves real money.
The failure mode here is not that AI defence is impossible. It is that the industry has been accidentally training attackers far harder than defenders, because attacking is easier to grade. Fixing that scoring problem is the work.
Operational takeaway: if your team is evaluating AI security tools, ask the vendor for blue-team benchmark scores, not just red-team demos. The demos always look good.
