What 300,000 Real-World Security Tests Taught One Company About AI Hacking Tools
Autonomous penetration testing has reached genuine scale. The hard lesson from running 300,000 tests is not about finding weaknesses. It is about knowing which ones actually matter.
Key points
- NodeZero has run more than 300,000 automated penetration tests, which are controlled simulated attacks on a company's own systems, since 2019.
- In one financial-services case, a single stolen password led to 586 critical security failures across 115 separate computer systems.
- A cloud environment was fully taken over without using any known software flaw; the attackers chained together weaknesses that existing tools had already spotted but never connected.
- Most organisations already have more security findings than they can act on; the real gap is knowing which findings an attacker would actually use.
- The source article was published by CSO Online as a product-perspective piece from Horizon3.ai, the company behind NodeZero.
For the past several years, hardly a month has passed without a new announcement promising that artificial intelligence can replace the human hacker in security testing. The claims are easy to make. The proof is harder to produce.
Horizon3.ai, the company behind the autonomous penetration-testing platform NodeZero, says it now has something most vendors in this space do not: a very large track record. More than 300,000 production tests run since 2019, across healthcare networks, financial institutions, manufacturers, and schools.
What that experience keeps teaching them, they say, is counterintuitive.
What is the actual problem AI security tools are solving?
Most organisations are not short of security alerts. They are drowning in them. Vulnerability scanners, cloud-monitoring dashboards, and exposure-management tools produce thousands of findings. Security teams spend enormous energy debating which ones are serious.
The trouble is that criminals do not think one finding at a time.
An attacker spots a small weakness, pairs it with a second small weakness, moves quietly through a network, and reaches something valuable. No single step along that path looks alarming in isolation. Together, they are devastating. Security tools that report each flaw independently, without showing how the flaws connect, give teams an incomplete picture.
Horizon3.ai calls the difference between severity and exploitability the central problem. Severity describes how bad a flaw looks on paper. Exploitability describes whether a real attacker could actually use it today, in your environment, to reach something that matters.
How does this play out in the real world?
The company offers three concrete examples from its test history.
| Environment | Starting point | Result |
|---|---|---|
| Financial services | One stolen password | 586 critical impacts across 115 systems, including three full domain takeovers |
| Cloud (Entra ID) | Known, already-flagged weaknesses | Complete tenant takeover, no software flaw required |
| Education | Initial access gained | Attacker paths reached systems never expected to be reachable |
The cloud case is worth pausing on. Entra ID is Microsoft's cloud-based identity system, the service that controls who can log into what. The testers achieved full control without exploiting a CVE, which is an officially catalogued software vulnerability with an assigned tracking number. Every weakness they used was already known. Existing tools had flagged it. Nobody had connected the dots.
Should security teams trust AI tools running inside live systems?
This is the question the article circles without fully answering, but it is the right one to ask. Running automated attack simulations against systems that real nurses, customers, or students depend on carries genuine risk. A misconfigured test could disrupt a hospital's patient records or take down a manufacturer's production line.
Horizon3.ai's argument is that safe, predictable operation in live environments is the hard engineering problem, and that it takes years of real-world runs, not benchmark demonstrations, to solve it. That is a reasonable position. It is also, naturally, a sales argument for their own platform.
Independent verification of those 300,000-test claims would be welcome. Until third-party auditors weigh in, treat the figures as directionally interesting rather than settled.
If your organisation uses any automated security-testing tool, ask the vendor for specifics: What safeguards prevent a test from disrupting live services? Can you limit the scope to particular systems? Who is notified if something goes wrong during a test? Those are practical questions any vendor should answer clearly.
