The AI framework you choose is also a security choice

Key points

• Across thousands of adversarial test runs, compromise rates ranged from 11.9% to 31.1% depending solely on which AI orchestration framework was used.
• The same underlying AI model was used throughout; only the framework wrapper changed.
• The four frameworks tested were CrewAI, LangChain, AutoGen, and SmolAgents.
• Framework design choices, such as how strictly tool calls are checked before execution, directly determine how much room an attacker has to work with.
• No major public comparison guide for these frameworks currently includes measured attack-success data.

Choosing between LangChain, CrewAI, AutoGen, and SmolAgents for an AI project is usually treated as an engineering question: which one has the best developer experience, the healthiest ecosystem, the cleanest way to link multiple AI agents together? A researcher publishing through CSO Online decided to ask a different question entirely. Does the framework you pick change how easily your AI agent gets broken into?

The answer, it turns out, is yes. By a wide margin.

What is an orchestration framework, and why does it matter?

Think of the AI model as an engine. The orchestration framework is everything around it: the gearbox, the brakes, the wiring that decides when the engine is allowed to act and when it must stop and check. It controls how the agent plans steps, when it calls an outside tool or service, how it stores information across a task, and how much it can do on its own before a human reviews the result.

The model does the thinking. The framework decides what the thinking is allowed to do.

That distinction turns out to matter enormously for security.

How was the test run?

The researcher built a testing harness that fired the same set of attack scenarios at AI agents built on each of the four frameworks. Attack types included tool call hijacking (tricking the agent into running the wrong command), memory poisoning (feeding false information into what the agent remembers), and delegated authority abuse (getting the agent to act beyond its intended permissions).

Crucially, the AI model stayed identical across every test. Only the framework changed. If frameworks were just neutral plumbing around the same model, the attack-success rates should have clustered together. They did not.

What did the numbers show?

The gap between best and worst was 2.6 times, produced entirely by the framework choice, with no change to the model or the attacks.

Why does the framework affect the attack rate?

Frameworks are not neutral. Each one makes specific design decisions that either close off attack paths or leave them open.

One concrete example: a framework that forces every tool call through a formal schema check, a structured set of rules that confirms the instruction is valid before it executes, gives an attacker far less room to sneak in a malicious instruction. A framework that lets the model call a tool directly from its own generated text, without that check, is easier to manipulate. That design choice was made by the framework's authors long before any customer's team touched it.

What should teams actually do?

If you are choosing a framework now, treat the security question with the same weight as the developer-experience question. Vendor feature pages tell you what a framework claims to do; they do not tell you how it holds up when someone tries to hijack a tool call.

If you have already shipped a system, test the specific setup you are running. Knowing where it sits on this spectrum tells you how urgently you need to harden it.

Do not assume a well-aligned AI model protects you automatically. The same careful model, wrapped in a looser framework, can end up with a significantly worse real-world attack rate than it would inside a tighter one. The model is not the whole attack surface. The framework is doing real security work, whether it was designed that way or not.