Your Incident Response Playbook Almost Certainly Does Not Cover AI Failures. That Is a Problem.
Seven in ten organisations have AI plugged into their core systems, yet most security teams are trying to handle AI breakdowns with tools built for a completely different kind of threat.
Key points
- Documented AI incidents rose 56.4% between 2023 and 2024, reaching 233 confirmed cases.
- The Epic Sepsis Model, deployed across hundreds of US hospitals, missed two-thirds of actual sepsis cases according to a 2021 JAMA Internal Medicine study, with no attack involved.
- Air Canada was held legally liable after its chatbot invented a discount fare policy and a customer relied on it.
Most security teams have an incident response playbook, the documented plan a company follows when something goes wrong digitally. The assumption that it covers AI systems is, among practitioners who've tested it, almost always wrong.
CSO Online published a detailed breakdown of this gap earlier this year, drawing on more than 14 years of hands-on experience across energy, banking, telecom and manufacturing. The picture it paints is uncomfortable.
Why does this matter to ordinary people?
Because AI failures already reach patients, travellers and job-seekers, not just IT departments. When a hospital's sepsis-detection model quietly lost useful accuracy, doctors received floods of false alarms. Nobody hacked it. Every dashboard showed green. It was doing exactly what it was built to do, just badly.
That distinction sits at the heart of the problem. There are two broad categories of AI failure. Model-originated failures cover degraded outputs, bias, and hallucinations (where an AI confidently states something untrue). Externally induced failures happen when an attacker deliberately corrupts the data a model sees. Most current incident-response frameworks treat these categories identically. They shouldn't, because the detection approach, the containment logic, and the legal exposure differ sharply between them.
The legal piece is no longer theoretical. Air Canada's chatbot invented a bereavement fare that didn't exist, a passenger relied on it, and a court held the airline responsible. In the US case Mobley v. Workday, a federal court accepted that an AI hiring platform could carry direct liability for bias in its decisions. Neither event resembled a security breach. Both ended as legal ones. If your legal team isn't on your incident response call sheet, the playbook was incomplete before any incident started.
The standard CIA triad, which stands for confidentiality, integrity and availability, also falls short here. When a chatbot fabricates a policy, nothing is stolen, nothing is altered by an intruder, nothing goes offline. Every traditional alarm stays quiet.
We covered the shutdown-authority gap in our 13 July story "Your AI risk register is a list, not a plan", and the pattern in the CSO Online analysis reinforces it.
Organisations ahead of this problem tend to have four things in place before any incident occurs. First, an inventory of every AI system in use, covering what data trained it and what it connects to. Second, a plain-language summary for each system that a responder can read at 2am, not a technical document written only for data scientists. Third, a data scientist named on the incident call list with the authority to interrogate the failing model in real time. Fourth, a pre-agreed threshold defining exactly when a misbehaving model gets switched off or replaced by a simpler fallback.
Should you worry?
If your organisation uses AI in customer service, hiring, clinical settings, or financial decisions, the honest answer is yes. Ask your team where the AI system inventory lives and who has authority to pull the plug. Get legal in the room before you need them.
The detection gap is real: average AI incident detection runs at 4.5 days. That's a long time for a model to quietly fail while every dashboard shows green.
