A 15-Minute Framework for Spotting What AI Systems Can Do Wrong
Security expert Adam Shostack built PHANTOM-B to help organisations find the risks hiding inside AI-powered software before those risks find them.
Key points
- Adam Shostack presented PHANTOM-B, a new AI security framework, at Black Hat USA 2026.
- A client's AI app, built with no formal security review, prompted the first real-world test of the framework, which surfaced meaningful risks in 15 minutes.
- PHANTOM-B stands for eight specific AI risk categories: Prompt injection, Hallucination, Anthropomorphization, Non-explainability, Training issues, Overreliance, Missing security engineering, and Bias.
- Industry bodies including OWASP, NIST, and Microsoft have published their own AI security guidance, but no single agreed standard yet exists.
- Experts say companies are deploying AI faster than their security teams can check it.
Some weeks ago, Adam Shostack opened an email from a client. One of their staff had built an AI-powered app, the kind assembled quickly using an AI coding tool with very little human review, and pointed it at real customer data. The client wanted to know what could go wrong. Fast.
Shostack gave himself 15 minutes.
By the end, he had a list of genuine risks, including hallucination (where an AI system confidently makes up false information) and bias (where the system's outputs systematically favour or disadvantage certain groups). Neither would have appeared on a standard security checklist.
What is PHANTOM-B and how does it work?
PHANTOM-B is a threat modelling framework, a structured checklist that security teams use to find weaknesses in a system before attackers do. Shostack built it specifically for software that uses large language models, or LLMs, the kind of AI engine behind tools like ChatGPT.
Each letter names a distinct risk:
| Letter | Risk | Plain meaning |
|---|---|---|
| P | Prompt injection | Criminals hide instructions inside text the AI reads, hijacking its behaviour |
| H | Hallucination | The AI produces confident but false outputs |
| A | Anthropomorphization | Teams trust the AI as if it thinks like a person, giving it more authority than it deserves |
| N | Non-explainability | The AI cannot reliably explain how it reached a decision |
| T | Training issues | Flawed or manipulated training data shapes bad outputs |
| O | Overreliance | Staff or systems depend on AI output without adequate checks |
| M | Missing security engineering | Old-fashioned software vulnerabilities that AI deployments still carry |
| B | Bias | Systematic unfairness baked into the model's responses |
Shostack presented the framework at Black Hat USA 2026. He is clear that PHANTOM-B sits alongside existing tools, not above them. The older STRIDE framework, a widely used security checklist for conventional software, still handles the rest of an application. PHANTOM-B handles the AI parts specifically.
Why does ordinary security checking fall short for AI?
Conventional software follows fixed rules. Feed it the same input twice and you get the same output. AI systems do not work that way. They interpret instructions written in plain language and generate responses that can vary each time, which makes predicting their behaviour much harder.
That unpredictability creates gaps that existing tools were never designed to catch.
Jeff Williams, founder of OWASP (the Open Worldwide Application Security Project, a non-profit that publishes free security guidance) and CTO of Contrast Security, puts it bluntly: "AI didn't break threat modelling. It exposed weaknesses that were already there."
Brian Glas, vice president at security consultancy CODIFIC and a project lead for the OWASP Top 10 for LLM Applications, adds another concern: weaknesses can chain together across AI systems. A poisoned document fed into an AI assistant, for instance, could alter its response and cause a connected tool to take a harmful action somewhere else entirely.
Should businesses worry about apps built without security checks?
Yes, and the problem is spreading. Shostack's 15-minute test was triggered by exactly the kind of software that security teams rarely get to review: a quickly built AI app handling real customer data, assembled by someone who wasn't a professional developer and deployed before anyone asked hard questions.
NIST, the US National Institute of Standards and Technology, recommends in its AI Risk Management Framework that threat modelling should be a continuous habit, not a one-off audit. Microsoft's own AI security guidance, first reported by CSO Online, makes a similar point: when AI systems can access data or trigger other tools, a single weakness can ripple outward in ways that are genuinely hard to trace.
The practical advice from Shostack is simply to start small. A 15-minute session will not catch everything, but it should answer one question: is the risk here acceptable, or does someone need to look harder?
"You make the experiments cheap," he says, "and when the experiment is cheap, you can run it repeatedly."
For anyone using an AI-powered product at work, the immediate step is straightforward: ask whether your organisation has reviewed what the tool does with your data, and whether anyone has formally checked what could go wrong if it produces wrong or biased results.
