The AI framework you choose is also a security choice
A researcher ran the same attacks against four popular AI agent frameworks and found the most vulnerable was 2.6 times more likely to be broken than the most resistant, using the identical AI model throughout.
Key points
- Across thousands of adversarial test runs, compromise rates ranged from 11.9% to 31.1% depending solely on which AI orchestration framework was used.
- The same underlying AI model was used throughout; only the framework wrapper changed.
- The four frameworks tested were CrewAI, LangChain, AutoGen and SmolAgents.
- Framework design choices, including how strictly tool calls are checked before execution, directly determine how much room an attacker has to work with.
- No major public comparison guide for these frameworks currently includes measured attack-success data.
Choosing between these frameworks for an AI project is usually treated as an engineering question: which one has the best developer experience, the healthiest ecosystem, the cleanest way to link multiple AI agents together? A researcher publishing through CSO Online decided to ask something different. Does the framework you pick change how easily your AI agent gets broken into?
The answer is yes. By a wide margin.
What is an orchestration framework, and why does it matter?
Think of the AI model as an engine. The orchestration framework is everything around it: the gearbox, the brakes, the wiring that decides when the engine is allowed to act and when it must stop and check. It controls how the agent plans steps, when it calls an outside tool or service, how it stores information across a task, and how much it can do autonomously before a human reviews the result.
The model does the thinking; the framework decides what the thinking is allowed to do. That distinction matters enormously for security.
How was the test run?
The researcher built an evaluation harness that fired the same attack scenarios at AI agents built on each of the four frameworks. Attack types included tool call hijacking (tricking the agent into running the wrong command), memory poisoning (feeding false information into what the agent remembers) and delegated authority abuse (getting the agent to act beyond its intended permissions). The AI model stayed identical across every test. Only the framework changed.
If frameworks were just neutral plumbing around the same model, attack-success rates should have clustered together. They didn't.
What did the numbers show?
| Framework | Compromise rate |
|---|---|
| Most resistant (lowest) | 11.9% |
| Second | not disclosed individually |
| Third | not disclosed individually |
| Least resistant (highest) | 31.1% |
The gap between best and worst was 2.6 times, produced entirely by framework choice, with no change to the model or the attacks.
Why does the framework affect the attack rate?
Frameworks aren't neutral. Each one makes specific design decisions that either close off attack paths or leave them open.
One concrete example: a framework that forces every tool call through a formal schema check, a structured set of rules that confirms the instruction is valid before it executes, gives an attacker far less room to sneak in a malicious instruction. A framework that lets the model call a tool directly from its own generated text, without that check, is easier to manipulate. That design choice was made by the framework's authors long before any customer's team touched it.
Our earlier story on memory poisoning attacks against AI platforms showed how that particular attack class can persist even after a patch; here, the framework itself is what determines whether the attack gets a foothold at all.
Should you worry?
Yes, if you haven't tested what you're actually running. Vendor feature pages tell you what a framework claims to do; they don't tell you how it holds up when someone tries to hijack a tool call.
If you're choosing a framework now, treat the security question with the same weight as the developer-experience question. If you've already shipped, test your specific setup. Knowing where it sits on this spectrum tells you how urgently you need to harden it.
Don't assume a well-aligned AI model protects you automatically. The same careful model inside a looser framework can end up with a worse real-world attack rate than it would inside a tighter one. The model isn't the whole attack surface. The framework is doing real security work, whether it was designed that way or not, and right now that fact isn't in any of the public comparison guides your team is probably reading.
