AI Red Teaming Grew Up. The Job Description Is Still Being Written.

When Microsoft stood up its AI red team in 2019, Ram Shankar Siva Kumar could fit every practitioner in the field onto a small sailboat. That era is over.

The problem is that the tools that replaced it aren't fully settled yet. When GPT-4 landed, Siva Kumar's team discovered their existing attack library against traditional ML systems simply didn't transfer. Everything had to be rebuilt. Methodologies. Tooling. Even the definition of the job itself.

That rebuild is still happening.

Probabilistic systems break deterministic testing

Traditional pen testing has a clean success condition: either the vulnerability is there or it isn't. AI doesn't work that way. The same prompt injection might succeed nine times out of a hundred, or ninety. Dane Sherrets, staff innovation architect at HackerOne, puts it plainly — security teams now have to answer not just whether a failure mode exists, but how often, under what conditions, and whether it reproduces reliably. That's a different kind of testing infrastructure entirely.

Pete Bryan, technical lead on Microsoft's AI red team, frames it as a consequence of probabilistic architecture. You can't run an eval once. You run it repeatedly, vary the conditions, and watch for risky outputs that only surface under specific load or phrasing patterns.

In practice, most organizations don't have the tooling or patience for that. The failure mode here is teams doing a single red-team pass before launch and calling it done.

The threat actor roster expanded

Classic red teams model nation-states and APT groups. AI red teams still do that — but Siva Kumar's team also explicitly models what he calls "a teenager with a potty mouth." That's not a joke about content moderation. It reflects something real: a large portion of the most significant jailbreaks discovered to date came from non-expert users with creativity and time, not from sophisticated offensive operators. Ian Swanson at Palo Alto Networks describes this as a push toward behavioral testing that covers security, safety, and brand reputational risk simultaneously.

The question shifts from can an attacker break in to can the system itself behave in ways that create harm — regardless of attacker intent.

Red team the whole car, not just the engine

Sherrets uses a car analogy that's worth keeping. The model is the engine. But production AI systems are everything bolted around it — the APIs, the downstream databases, the customer records, the internal workflows those systems can reach. Weaknesses emerge from component interactions, not just the model weights.

The Air Canada case makes the stakes concrete. The airline's customer service chatbot fabricated a bereavement refund policy. A customer relied on it. The airline ended up in court. No attacker. No CVE. The system just behaved incorrectly and the organization owned the output.

As AI gets embedded into customer service, HR, and internal ops, that failure mode scales.

Microsoft's team now includes a psychologist, a linguist, and a bioweapons specialist — expertise that would have read as bizarre on a traditional SOC org chart. That expansion tracks with a broader shift: AI red teaming has absorbed safety, misinformation risk, and sociotechnical harm evaluation alongside the CIA triad it started with.

Washington noticed briefly. Biden's 2023 executive order formally defined AI red teaming and required safety testing for frontier models before deployment. Trump revoked it. Standards are now largely voluntary.

Operational takeaway: If your AI red team is only testing the model, you're not red teaming the system.