When AI Does What It's Told, For the Wrong Person

Key points

• Noma Labs researcher Sasi Levi published findings describing a technique called "workflow identity hijacking," in which an outsider can trigger privileged AI actions through an unauthenticated entry point such as a support inbox or web form.
• The flaw doesn't require manipulating the AI model itself; the model behaves exactly as designed, but executes actions using the workflow creator's high-level permissions rather than the requester's.
• Because the resulting activity runs through valid credentials and trusted automation, it can blend into routine system logs without triggering standard security alerts.
• Cybersecurity researcher Vibhum Dubey described the pattern as a "confused deputy" problem, a known class of flaw where a trusted system is tricked into misusing its own authority on behalf of someone who shouldn't have it.
• Google's Threat Intelligence Group has separately noted that criminals are beginning to use similar multi-step AI workflows to carry out attacks autonomously.

---

What actually happened here?

Researchers at Noma Labs have identified a way that ordinary messages sent to a company's AI-powered tools can quietly trigger actions that only senior employees should be able to authorise.

Many companies now use AI-driven automation: software pipelines that read incoming requests and carry out tasks automatically, pulling financial records, updating databases, sending files. These pipelines often accept input from public-facing channels such as a customer support inbox, a shared document, or a web form. Anyone can drop a message in.

When the AI pipeline picks up that message and acts on it, it does so using the identity and permissions of whoever built or owns the workflow, often a developer account or high-level service account with broad access. The pipeline never checks whether the person who sent the original message was actually allowed to make that request.

Sasi Levi, lead researcher at Noma Labs, put it plainly: "The core failure is that the requester had no authority to make that request."

Why is this so hard to catch?

Standard security tools look for suspicious content. This attack sends none.

Levi's report includes a telling illustration. Imagine two people ask the same AI tool for the same financial report: a chief financial officer, and a random outsider. The message is word-for-word identical. A standard security filter sees two harmless prompts and flags neither. But one person is allowed that information; the other isn't.

Because the pipeline uses its own valid credentials to fetch the data, every step looks legitimate. No stolen password, no malicious code, no obvious red flag. Dubey described the detection challenge bluntly: "The individual events can look completely legitimate."

Spotting the problem, he said, requires correlating who sent the original message, which identity the system used downstream, what resources were accessed, and what action was finally taken. Most organisations aren't set up to join those dots automatically.

What does this mean for companies using AI tools?

The risk grows as AI automation expands. A pipeline that quietly fetches a financial report for an unauthorised requester today could, depending on its configuration, send files or update records tomorrow. That's not a hypothetical edge case; it's the design.

This fits a pattern our authorization coverage has tracked since May: as we reported on 26 August in "Passing the Login Test Does Not Mean You Let In the Right Person", confirming that a request arrived through a legitimate channel is not the same as confirming the requester had the right to make it.

Levi's report argues that fixing this requires moving security controls out of the AI model and into the surrounding infrastructure. Every step in a workflow should verify the original requester's permissions before acting, not just trust the identity of the system running it.

Dubey offered a practical test: if a low-privileged user and an administrator send the same request and receive the same result, that's where the investigation should start.

Common questions

Does this affect me as a customer or user of a business?

If a company you deal with uses AI-powered tools to handle requests or manage data, a gap like this could mean an outsider accessed information that should have stayed private. The research describes a class of design flaw, not a single breach, so there's no public list of affected companies.

What should organisations do right now?

Levi's report recommends that every AI workflow carry the original requester's identity all the way through to the point where an action is executed, and check permissions there, not just at the front door. Reviewing which service accounts your AI pipelines run under, and what access those accounts hold, is a sensible immediate step.