Most AI Models Fail New Malware-Investigation Test Inspired by Nuclear Sabotage

Key points

• SentinelOne, a cybersecurity company, built a new benchmark to test whether AI models can fully investigate a piece of malware from first clue to final verdict.
• The benchmark is built on the Fast16 case, malware linked to attacks on industrial control systems including nuclear facilities.
• Most leading AI models tested could not sustain a complete investigation, failing partway through the analytical chain.
• No AI model is currently reliable enough to run a malware hunt without a human analyst checking its work.

SentinelOne has built a test that most of the world's best-known AI systems quietly fail.

The test is called the Fast16 benchmark. It asks an AI to do what a senior malware analyst does: take a suspicious piece of software apart, piece by piece, until the whole picture is clear. Not just spot that something looks bad. Actually finish the investigation.

Most models cannot.

What is the Fast16 case, and why does it matter?

Fast16 refers to a set of malware samples connected to attacks on industrial control systems, the software that runs power grids, nuclear facilities, and water treatment plants. The name has circulated in threat-intelligence circles as shorthand for a particularly complex class of attack.

SentinelOne chose it precisely because of that complexity. Investigating this malware isn't a one-step task. It requires an analyst to hold many threads at once, pivot from one clue to the next, and reason about what the attacker was ultimately trying to do. That chain of reasoning is where AI models break down. Our coverage of industrial control systems goes back to at least 16 July, when we reported on the decades-old code still running inside factories and power grids, and the fragility there makes the AI reliability question more than academic.

Why do AI models struggle with this?

They lose the thread. An AI model works well on individual, self-contained questions. Ask it to explain one function in a piece of code, and many models do a decent job. Ask it to link that function to a command-and-control server, connect that server to a known group, and judge whether the payload could disable industrial equipment, and the model starts to drift.

Security researchers call this "sustained reasoning." Humans do it naturally during an investigation. Current AI systems, it turns out, don't.

This matters because a growing number of security vendors are marketing AI as a tool that can handle investigations with minimal human oversight. SecurityWeek, which first reported on SentinelOne's research, noted the benchmark lands at a moment when those claims are facing real scrutiny.

Should security teams be worried about relying on AI?

Yes, if they're relying on it unsupervised. No tool that can't finish a thought is safe to leave alone.

That's not a reason to discard AI from security workflows. Analysts already use it to speed up early triage, the first rough sort of which alerts deserve a closer look. It earns its place as a first-pass assistant, not a replacement for the human who connects the final dots.

For critical infrastructure, hospitals, utilities, the consequences of a gap here aren't theoretical. A tool that fails this benchmark, without a person checking its work, leaves an opening a sophisticated attacker can walk through.

SentinelOne hasn't published a full list of which models passed or failed. That detail, when it arrives, will tell a clearer story about which AI vendors' confidence is actually warranted.