The Cheapest Way to Beat an AI Security System Is to Read Its Rulebook
Confidence in autonomous hacking tools has collapsed. A researcher says the real problem runs deeper: the governance rules we write to keep AI security systems safe can become a weapon in an attacker's hands.
Key points
- The share of organisations willing to rely on autonomous penetration testing, meaning AI systems that hunt for security weaknesses, dropped from 29% in 2025 to 9% in 2026, according to a Cobalt report cited by Dark Reading.
- Independent researcher Burak Oktenli argues the deeper problem is "cap weaponisation": attackers exploiting the written governance rules around an AI system rather than its code.
- An attacker who understands a system's recovery rules can drain it of authority over eight weeks without ever touching its software.
- The fix is cross-layer auditing: computing the system's authority in one place, enforcing it somewhere independent, and comparing both records so any manipulation shows up as a mismatch.
- Security leaders should red-team the rulebook itself, not just the technical systems it governs.
Why are organisations losing faith in AI security tools?
Fewer than one in ten organisations now trust autonomous security tools to work without close human supervision, down from nearly one in three a year ago.
The easy reading is that the technology overpromised. Oktenli thinks something more specific is happening: attackers have learned that it's cheaper to work around the rules an AI system's owners wrote than to crack the system open.
How does attacking the rulebook actually work?
Every AI security system ships with governance rules. These set an authority ceiling (the maximum action the system can take on its own), recovery protocols (instructions to step back when something looks wrong, then resume slowly), and an audit log (a record of every decision).
Those rules are published in product sheets and compliance standards. Anyone can read them.
Oktenli describes a scenario he calls cap weaponisation: an attack aimed at the governance layer rather than the software underneath. Picture a defensive system at the edge of a company's network, working exactly as designed. Over eight weeks, nothing in its daily logs looks alarming.
Yet the system drifts into a state where humans must approve nearly every action, crippling its speed. No exploit was fired. The attacker read the recovery rule (step back fast, recover slowly) and ran short, cheap bursts of low-level probing to keep knocking the system down before it could climb back up. The probes cost almost nothing, and the damage is free because the system is just following its own instructions.
What are the other openings an attacker could use?
The slow-drift attack is one of four routes Oktenli maps through the governance layer.
| Attack route | What it does |
|---|---|
| False inputs | Feeds the system fake readings so its authority calculation sees conditions that do not exist |
| Slow configuration drift | Slips under the authority ceiling gradually, without a clean exploit |
| Fake threat-clear signals | Tricks the system into recovering authority it has not earned |
| Weakest coalition node | In shared or coalition setups, targets the softest participant to drag down collective decisions |
None of these requires breaking encryption or finding a zero-day, meaning a software flaw the maker did not yet know about. They require patience and a copy of the specification.
What should security teams actually do?
Oktenli's answer isn't to hide the governance rules. Hiding them kills the transparency that makes the system trustworthy. The fix is cross-layer consistency: compute the system's authority in one place, enforce the ceiling independently, record both, then have a review process compare them. An attacker who manipulates one layer without touching the others produces a mismatch the audit can catch.
Three practical steps follow. Root the authority-ceiling enforcement in hardware with its own independent record, so software can't silently override it. Review system behaviour over long windows, not just daily logs; the eight-week drift Oktenli describes is invisible day to day but obvious across a quarter. Finally, pay red teamers, security professionals hired to attack your own systems, to assault the rulebook and not just the code.
We've covered the governance gap in autonomous systems before: our 20 July report "Your AI Is Moving Faster Than Your Security Team Can Follow" found that boards are greenlighting AI at speed while the tools to verify those systems' behaviour haven't kept up. Oktenli's argument lands in the same gap, from the attacker's side. That's the thing worth watching here: the governance layer isn't a solved problem teams can file away. Adversaries are already treating it as an attack surface, and most red-team programmes aren't testing it at all.
