Treat the Model Like a Threat: Why AI Agent Security Needs a Systems Overhaul

Enterprises relying on model-side fixes to secure AI agents are solving the wrong problem, according to a paper published this month by researchers at Google, the University of California San Diego, and the University of Wisconsin-Madison, among other institutions. The core argument is simple: once an agent has access to memory, APIs, browsers, and execution environments, semantic guardrails become largely decorative.

The paper, whose authors include Mihai Christodorescu, Earlence Fernandes, and Somesh Jha, draws an explicit comparison to operating-system design. An OS treats every process as untrusted and enforces controls outside it. AI agents, the authors argue, should be treated the same way. "The AI model powering the agent must be treated as an untrusted component," they wrote, with security properties expressed and enforced at the system level, not baked into prompts or alignment tuning.

Predictably, the evidence they assembled is not encouraging for vendors currently selling model-centric safety as a solution.

The researchers analysed eleven real-world attacks on AI agents, mapping each to violated security principles. The catalogue included a data-exfiltration flaw in the ChatGPT macOS application, an exfiltration vulnerability in Claude Code, a similar issue in Microsoft Copilot, and the AgentFlayer attack on Cursor — delivered through a malicious Jira ticket. Every single one of the eleven attacks violated the secure information flow principle. Most also violated least privilege.

From decades of systems security research, the authors distilled five principles they say agentic systems must follow: least privilege, tamper resistance of the trusted computing base, complete mediation, secure information flow, and accounting for the human as a weak link. (That last one is, as always, the principle most organisations prefer not to dwell on.)

They were equally direct about stacking additional ML models as a defensive measure. "Merely stacking ML models does not constitute true defense-in-depth," the paper said, noting that guard models "often share the same statistical failure modes as the primary agents they monitor." A guard model trained on similar data to the model it watches is not a meaningful control boundary.

The authors proposed three concrete mechanisms, each tied to an open research problem. First: separating instructions from data, which current language models do not do — both arrive as an undifferentiated stream of tokens. Second: verifiable least-privilege policy generation, complicated by the fact that agent security policies are written in natural language and shift mid-task. Third: information flow control, since tracking how sensitive data actually moves through a model remains unsolved.

So the practical situation is that enterprises are deploying systems whose internal reasoning is opaque, whose policies are fluid, and whose defensive layers may fail in correlated ways — while treating prompt injection as a content problem rather than a workflow-execution and systems-integrity problem.

A separate paper published last week approached the visibility gap from a different angle, proposing an "agentic detection and response" (ADR) framework designed to inspect agent reasoning flows, prompt chains, and dynamic tool execution that conventional endpoint detection and response platforms cannot see. In a reported production deployment monitoring more than 10,000 AI-agent sessions daily across roughly 7,200 hosts, the framework identified hundreds of credential-exposure incidents across 26 attack categories. On a benchmark the team called ADR-Bench, the system achieved a 67% detection rate with zero false positives, outperforming Meta's LlamaFirewall by two to four times on F1-score. On AgentDojo, it detected all attacks with three false alarms across 93 tasks.

The research does not offer a finished solution. It offers a more honest framing of the problem — which, given how the past two years of enterprise AI-security marketing have gone, is a reasonable place to start.