Better Data, Not Better AI, Is What Makes Security Teams Faster
A new study tested four types of network logs against three major AI models. The data source mattered far more than which AI you picked.
Key points
- High-quality network logs improved AI-assisted security investigation accuracy to 95.2%, compared with 25.8% for basic connection records.
- Investigation time dropped to 14.7 minutes with rich logs, versus 27 minutes with basic flow data.
- The research used simulated versions of two real espionage campaigns: Volt Typhoon and Salt Typhoon, both tracked as Chinese state-linked intrusion clusters.
- Model upgrades alone can't fix poor data: swapping AI providers made far less difference than improving what the AI was shown.
- Critical incident visibility was five times higher with enriched logs than with basic connection records.
Security teams at large organisations are rushing to deploy AI tools that sort through thousands of alerts, flag the suspicious ones, and draft incident reports automatically. The promise is real. But a research project published this year suggests most teams are asking the wrong question when they shop for AI: which model is best? The research's answer is that it barely matters, as long as your underlying data is poor.
How did the researchers test this?
The study, called Provably Better Data, ran a controlled experiment using two well-known attack scenarios. One was built around Volt Typhoon, a cluster tracked by Microsoft and other vendors as Chinese state-sponsored, known for quietly embedding itself inside critical infrastructure. The other drew on Salt Typhoon, a separate Chinese espionage group that has targeted telecommunications providers. Our earlier coverage of Salt Typhoon, first published 17 August, noted a recurring theme: visibility gaps, not model choice, drive investigation failures.
The researchers fed four types of network logs, records of what computers did and said to each other, into three AI models: Anthropic Claude Opus 4.6, Google Gemini Pro 3.1 Preview, and older versions of both. Every model received identical instructions. Only the log quality changed.
The four types ranged from enriched logs that decode individual network protocols in fine detail, down to NetFlow records: basic connection summaries that note two computers talked but not what they said.
What did the numbers show?
The gap was stark.
| Data source | CTF accuracy | IR evidence coverage |
|---|---|---|
| Corelight enriched logs | 95.2% | 90.3% |
| Firewall logs | 58.3% | 61.3% |
| Snort 3 IDS alerts | 39.4% | 21.2% |
| NetFlow records | 25.8% | 30.9% |
For critical investigation questions, enriched logs let models answer 91.7% of mandatory questions. Basic NetFlow records supported just 18.3%.
Speed told the same story. Rich logs produced a completed investigation in 14.7 minutes. NetFlow logs took 27 minutes, because the AI kept hitting dead ends and retrying. Worse data nearly doubled the time.
One example illustrates why. Investigators asked the AI to identify a computer by name from its network address. Enriched logs contained the answer directly: a machine called FINANCE01. Basic firewall logs could see that a certain security protocol had been used, but had never recorded the actual computer name. The AI can't guess what was never written down.
Does the AI ever make things up?
Rarely, when instructed correctly. The researchers told models to mark unanswerable questions as unanswerable rather than guess. Hallucinations, meaning confident but fabricated answers, stayed at zero for three of the four data sources. Snort 3 alerts produced an average of 1.9 fabricated answers per run, likely because partial information tempted the model to fill gaps.
Structured instructions plus complete data keeps AI grounded. That's the practical lesson.
Should you worry about which AI vendor you're using?
Probably less than you think. This research, first reported by CSO Online, is consistent with what we argued in August: the bottleneck in AI-assisted security isn't the model, it's what the model sees. Richer network telemetry, the detailed logs that capture what protocols did and what data moved, should come before another AI licence renewal.
For most employees, the finding is reassuring: AI security tools, when fed good data, investigate incidents faster and with fewer mistakes. The catch is that good data requires infrastructure investment that's invisible to most staff. If your security team is asking for budget to upgrade its logging systems, this research gives them solid ground to stand on. Watch whether vendors start competing on log depth rather than model benchmarks. That shift would mean the lesson has landed.
