AI Security Scanners Are Drowning Teams in False Alarms, and the Fix Isn't More AI
More than 60% of flagged security flaws are noise. A researcher testing a dozen tools says AI models make the problem worse, not better, because they lack the context to tell a real threat from a ghost.
Key points
- More than 60% of vulnerabilities flagged by application-scanning tools in 2024 tests were false positives, unreachable code, or low-severity issues, according to Pixee CTO Arshan Dabirsiaghi.
- The Forum of Incident Response and Security Teams estimates the total number of formally tracked software flaws will rise 50% this year.
- One study found that 62% of open-source software libraries are never actually loaded when a program runs, meaning most flagged flaws sit in code nobody uses.
- Large language models (AI systems trained on huge amounts of text and code) flag different vulnerabilities on separate scans of the same code, making their output unreliable.
- Dabirsiaghi plans to present full test results at the Black Hat USA security conference in August 2025.
Security teams already spend most of their day chasing problems that turn out not to be problems. Now the tools meant to fix that are making it worse.
Arshan Dabirsiaghi, chief technology officer and co-founder of Pixee, an AI-focused application security company, tested more than a dozen software-scanning tools and found that six in ten alerts were junk: false positives (warnings about flaws that do not actually exist), flaws sitting in code the running application never touches, or issues so minor they barely matter. He plans to detail those findings at Black Hat USA in August.
Why does this matter to ordinary people?
Every app on your phone and every website you use runs on software. That software has bugs, some harmless and some dangerous. Security teams are supposed to find and fix the dangerous ones before criminals do. When the tools those teams rely on cry wolf constantly, the genuinely dangerous flaws get lost in the noise, and your data is at greater risk.
The failure mode here is simple: teams either automate everything and drown in a flood of low-quality patch requests, or they assign a human to check every single alert. Dabirsiaghi puts the share of alerts that represent a real, reachable, exploitable flaw at around 8%. Nobody has enough staff to manually review 100 warnings to find the eight that matter.
AI was supposed to change that arithmetic. It hasn't.
Large language models (AI systems trained on enormous libraries of text and code) struggle because they work without context. They see a line of code that looks risky and flag it, without knowing how the wider application actually runs in production. Worse, they are inconsistent. Run the same scan twice and the model may call the same flaw a false positive one time and a genuine vulnerability the next. "LLMs are dumb," Dabirsiaghi says bluntly. "It'll just say there's a vulnerability on line X and go fix it even if it's not real."
Over-fixing is its own problem. Patching code that was never broken can introduce new bugs, slow the application down, and erode the engineering team's trust in the security process entirely.
The practical fix, Dabirsiaghi argues, is reachability analysis, which means checking whether the vulnerable code is ever actually executed when the software runs. Research backs this up: one study found that 62% of open-source libraries are never loaded at runtime. Another found that of the code in Java applications (Java being one of the most widely used programming languages in business software), only 12% of the open-source portions are ever called. Cutting alerts to only the code that actually runs would slash the workload dramatically.
Good triage also needs three layers of context: who the organisation is, how its systems are built, and what the surrounding code actually does. A flagged use of MD5 (an older method of scrambling data for storage) might be critical if it is protecting passwords, or completely irrelevant if it is being used for an unrelated internal task. Without that context, an AI model defaults to a generic severity rating that helps nobody.
The Forum of Incident Response and Security Teams estimates the number of formally catalogued software flaws, each assigned a CVE (Common Vulnerabilities and Exposures) identifier, will grow 50% in 2025. More flaws, noisier tools, and no extra headcount is a bad combination.
One operational takeaway: reachability filtering is not glamorous, but it is the fastest way to cut your real alert queue without adding headcount.
