What 300,000 Real-World Security Tests Taught One Company About AI Hacking Tools
Autonomous penetration testing has reached genuine scale. The hard lesson from running 300,000 tests isn't about finding weaknesses. It's about knowing which ones actually matter.
Key points
- NodeZero has run more than 300,000 automated penetration tests, which are controlled simulated attacks on a company's own systems, since 2019.
- In one financial-services case, a single stolen password led to 586 critical security failures across 115 separate computer systems.
- A cloud environment was fully taken over without using any known software flaw; the attackers chained together weaknesses that existing tools had already spotted but never connected.
- Most organisations already have more security findings than they can act on; the real gap is knowing which findings an attacker would actually use.
- The source article was published by CSO Online as a product-perspective piece from Horizon3.ai, the company behind NodeZero.
For the past several years, hardly a month has passed without a new announcement promising that AI can replace the human hacker in security testing. The claims are easy to make; the proof is harder to produce.
Horizon3.ai, the company behind the autonomous penetration-testing platform NodeZero, says it now has something most vendors in this space don't: a large track record. More than 300,000 production tests run since 2019, across healthcare networks, financial institutions, manufacturers. We've been following the company since its $250 million funding round on 3 August.
What that experience keeps teaching them, they say, is counterintuitive.
What is the actual problem AI security tools are solving?
Most organisations aren't short of security alerts. They're drowning in them. Vulnerability scanners and cloud-monitoring dashboards produce thousands of findings; security teams spend enormous energy debating which ones are serious.
The trouble is that criminals don't think one finding at a time.
An attacker spots a small weakness, pairs it with a second, moves quietly through a network, and reaches something valuable. No single step looks alarming in isolation. Together, they're devastating. Tools that report each flaw independently give teams an incomplete picture, a point our 5 August piece on attack-path security covered in detail.
Horizon3.ai frames the central problem as the difference between severity and exploitability. Severity describes how bad a flaw looks on paper. Exploitability describes whether a real attacker could use it today, in your environment, to reach something that matters.
How does this play out in the real world?
The company offers three concrete examples from its test history.
| Environment | Starting point | Result |
|---|---|---|
| Financial services | One stolen password | 586 critical impacts across 115 systems, including three full domain takeovers |
| Cloud (Entra ID) | Known, already-flagged weaknesses | Complete tenant takeover, no software flaw required |
| Education | Initial access gained | Attacker paths reached systems never expected to be reachable |
The cloud case deserves attention. Entra ID is Microsoft's cloud-based identity system, the service that controls who can log into what. The testers achieved full control without exploiting a CVE, which is an officially catalogued software vulnerability with an assigned tracking number. Every weakness they used was already known. Existing tools had flagged it. Nobody had connected the dots.
Should security teams trust AI tools running inside live systems?
This is the question the article circles without fully answering, and it's the right one to ask. Running automated attack simulations against systems that hospital staff or students depend on carries genuine risk. A misconfigured test could disrupt patient records or take down a production line.
Horizon3.ai's argument is that safe, predictable operation in live environments is the hard engineering problem, and that it takes years of real-world runs, not benchmark demonstrations, to solve it. That's a reasonable position. It's also, naturally, a sales argument for their own platform.
Independent verification of those 300,000-test claims would be welcome. Treat the figures as directionally interesting until third-party auditors weigh in.
Vendors of any automated security-testing tool should be able to answer three practical questions: what safeguards prevent a test from disrupting live services, can scope be limited to particular systems, and who gets notified if something goes wrong. Any vendor that can't answer clearly isn't ready for production.
