Your Security Team Is Flying Blind on AI. Here Is Why.
The tools built to catch hackers and bad code were designed for a world where humans made every decision. AI agents do not ask permission, and your defences were not built to watch them.
Key points
- AI agents, meaning software programs that act on their own without waiting for a human to approve each step, are moving from lab experiments into live business systems faster than most security teams can track.
- Existing security tools, including code scanners and network filters, cannot detect "prompt injection", where a criminal slips hidden instructions into a live AI system to make it behave maliciously.
- Models pulled from public online libraries can arrive poisoned with altered behaviour, yet most companies apply no equivalent of the signing checks they already use for regular software.
- Every AI agent needs its own digital identity, scoped permissions, and an audit trail, requirements that most corporate identity systems do not yet cover.
- A whitepaper co-authored by Broadcom and PlatformEngineering.org, covered by CSO Online, argues the engineering team that runs the underlying platform is now a frontline security function.
What gap are we actually talking about?
For the past decade, companies got smarter about catching security problems early. Scan the code before it ships. Block suspicious logins. Check the software before it runs. That approach worked when a human being was behind every action.
AI agents break the model entirely.
An AI agent is a piece of software that reads data, makes decisions, calls other systems, and completes tasks, all without stopping to ask a person. It consumes APIs (the connectors that let software talk to other software) directly. It may touch your customer records, your billing systems, and your internal documents in a single automated chain, often with no human in the loop at all.
Your existing tools were not built for that.
How could criminals actually cause harm here?
Four attack types matter, and none of them show up on a standard security dashboard.
First, prompt injection. An attacker hides malicious instructions inside content the AI reads, a document, a support ticket, a webpage. The AI follows those instructions as if they were legitimate. No code scanner sees this because there is no code to scan.
Second, model poisoning. Companies often pull pre-built AI models from public online libraries. If a model has been tampered with before download, and there is no equivalent of a wax seal on the package, the company deploys the corrupted version without knowing.
| Attack type | What it means in plain English | Current tools that catch it |
|---|---|---|
| Prompt injection | Hidden commands trick the AI mid-task | None |
| Model poisoning | Tampered AI model pulled from a library | None |
| Inference data leaks | AI response reveals private customer data | Rarely, if ever |
| Shadow AI sprawl | Staff use unapproved AI tools outside controls | None |
Third, inference data leaks. When the AI answers a question, its answer may contain private customer information or internal secrets. Standard data-loss-prevention tools, which usually watch email and file transfers, do not watch AI conversation logs.
Fourth, shadow AI. Individual teams download and run their own AI tools outside any official approval process. Governance simply does not reach them.
What does the fix actually look like?
The argument from the Broadcom and PlatformEngineering.org paper is that no developer-side checklist closes these gaps. The fix has to live in the platform, meaning the shared infrastructure the whole company runs on.
In practice that means four things: a verified library of approved AI models with a clear chain of custody; filters that strip out injected commands before they reach the AI; data boundaries that mask private information inside AI pipelines automatically; and a full, continuous log of every decision an AI agent makes.
On identity, specifically, every AI agent needs its own account with limited permissions, just as an employee does. Most corporate identity systems have never been asked to manage a non-human account that calls hundreds of APIs per minute.
If your platform team cannot answer basic questions about how agent permissions are scoped or where the audit logs live, that is the gap to fix first.
Get your platform engineering lead in the room before the next AI workload ships to production.
