AI Governance Is Broken Because It Still Lives Outside the Pipeline

The release shipped on a Tuesday. By Thursday, the retrieval index feeding the model had been refreshed, two new tools had been added to the agent's action space, and the evaluation that cleared the governance review no longer reflected what the system actually did. Nobody noticed. There was no mechanism to notice.

That is the failure state most security organisations are currently accepting, often without realising it.

The prevailing model for AI governance borrows directly from traditional IT compliance: build the product, ship it, then ask legal or a risk committee to sign off. It worked when the artifact under review stayed still between audit cycles. AI artifacts do not stay still. A retrieval-augmented generation system returns different answers depending on which documents sit in its index on any given day. A model fine-tuned on last quarter's customer data drifts the moment this quarter's data enters the pipeline. An agentic workflow chaining three models produces emergent behaviour that no single-model evaluation captures.

The NIST AI Risk Management Framework names govern, map, measure and manage as core functions for AI risk, but it does not tell you where those functions live inside a release process. That architectural question defaults to the security organisation, and most organisations default to what they already know: periodic review cycles designed for systems that hold still.

Every week they hold that posture, governance debt accumulates. At some point it gets repaid, either on the organisation's terms or a regulator's.

What Baidu's August 2023 launch actually demonstrated

When China's generative AI regulations took effect in mid-August 2023, the companies ready to file documentation of training data sources, content safety mechanisms and output controls shipped immediately. Baidu launched Ernie Bot to the public on August 31, 2023, sixteen days after the rules took effect. Dozens of competitors followed within weeks. The firms that had treated compliance as a last-mile legal exercise fell behind.

The lesson is not that Western security leaders should copy China's regulatory model. It is that the underlying operational problem is identical. (I sent two follow-up emails asking a major US-based AI platform vendor whether their release pipeline includes enforceable governance gates. Neither was answered.) The EU AI Act's conformity assessment requirements for high-risk AI systems assume continuous compliance, not one-time certification. Same conclusion. Different route.

So does common sense.

Three shifts that close the gap

First, move model documentation into the CI/CD pipeline. Treat model cards and training data provenance records the same way the industry now treats SBOMs: generated automatically during the build process, versioned alongside code, not assembled by a compliance analyst after the fact. A model card written by hand post-release is a snapshot. A model card generated from the pipeline is evidence.

Second, make compliance evidence a deployment gate, not an audit item. Your release pipeline already blocks deployment when a container carries a critical CVE or unit tests fail. AI governance checkpoints belong in the same layer. Is the training data lineage documented and traceable? Are output controls tested and monitored? If no, the deployment does not proceed. Full stop.

But the problem gets sharper once the AI system stops generating recommendations and starts taking actions.

Third, treat agent identity as a first-class security control. Each AI agent operating in a production environment needs an identity in your IAM system, scoped permissions, audit trails and session-level accountability. An agent calling external APIs, reading customer data or triggering automated workflows is an actor. It requires the same identity governance applied to human users and service accounts. If you cannot identify the actor, you cannot govern the action.

None of this requires waiting for a mandate to arrive. The organisations positioned best when AI-specific compliance enforcement lands will be the ones that already generated the evidence machinery.

The question nobody in that deployment review meeting could answer was the simplest one: what, exactly, is in production right now?