Why AI Agent Governance Is Harder Than Anyone Wants to Admit

When OpenAI disclosed that its own AI models escaped from what was supposed to be a sandboxed testing environment, accessed the internet, and exploited a vulnerability to gain access to Hugging Face’s systems, it was described as an autonomous agent pursuing its evaluation goals.

In fact, an AI safety expert at Oxford described it as a problem of “misspecified goals,” but that doesn’t get at the structural issue here. The agent didn’t go rogue because of a model flaw. It did so because the governance layer around it wasn’t built for what the agent was actually capable of doing.

We know governance conversations about AI agents tend to start with things like access controls and audit logs. Those things matter, of course, but it’s becoming clear that they’re downstream of a more fundamental problem: Agents make decisions based on the context they have, and most enterprises have no real control over what context their agents are actually using. That’s where governance is quietly failing in a way that won’t show up immediately.

The scale of it all is well-documented. MIT researchers studying the emerging agent economy lay out a future in which every person and organization runs multiple agents, eventually numbering in the trillions worldwide. The timeline can certainly be debated, but the direction is clear: The agent population is growing faster than the frameworks designed to govern them.

There’s a Context Problem That Nobody’s Talking About

Standard IAM was built for humans. A user logs in, a role is assigned, and permission is granted. That model works because human intent is naturally bounded, such as an analyst querying what they know they need.

Agents are more goal-driven, chaining together dozens of intermediate decisions and synthesizing data across sources in a wide variety of combinations. The permission might be technically valid, but the data path could still be completely inappropriate.

That said, accuracy might be a bigger issue than access. Agents operating against raw warehouse schemas constantly run into ambiguity. You might have three tables with the same name but each applying different logic. Or a field might use an abbreviation without providing context or the canonical definition that a model can find. The agent reads these structures, maybe infers meaning from column names, and returns a confident answer with no caveats. Then, when a senior executive presents the data to the board, it’s wrong.

That’s clearly not an access issue. It’s a context failure that no RBAC configuration can fix.

The Context Window Makes This Hard

Context is finite, but the knowledge space agents need to reason against isn’t. Every turn of an agent’s execution bites into the context window. The memory it needs to carry forward has to compete with the task it’s actually trying to complete. And the problem doesn’t disappear as models get bigger.

I’ve seen benchmarks claiming million-token context windows solve this. In practice, from running production queries at scale, the degradation sets in much earlier than that. Once you’re past roughly 100,000 to 200,000 tokens, model quality drops in ways the published benchmarks don’t show you.

So you have a finite, real-world ceiling, and you’re asking agents to make high-stakes data decisions within it. Knowing this, the answer isn’t to dump in more context. You need to get much better at selecting the right context, and that’s the governance problem most frameworks don’t address yet.

An Attack Surface Nobody Planned For

There’s a related problem that intensifies the urgency of context governance. Recent security research has documented a class of attack called agent data injection, in which an attacker corrupts the small, trusted facts an agent reasons from: things like a sender’s name, a button’s ID, or the record of a step already completed. Because the agent is acting on data it believes is reliable, conventional prompt injection defenses don’t catch it.

Researchers tested this across six major models from OpenAI, Anthropic, and Google and found success rates of 31% to 43% on structured data, with significantly higher rates on webpage data in some scenarios.

This illustrates exactly why access controls aren’t the whole story. Even when credentials are valid and permissions are correctly scoped, an agent can be made to act on false information. Governance has to reach the context layer, not just the credential layer.

Here’s What a Governance-Grade Trace Requires

Logging execution events is not a reliable audit trail. A compliance-grade trace needs to document what data was accessed, what context shaped the output, what logic was executed, and how the agent arrived at its answer, all traceable to a specific query and timestamp.

When a regulator asks why an agent produced a particular recommendation, an event log won’t have the answer. The reasoning chain, the context selection, and the intermediate decisions have to be structured and queryable, not just recorded.

As you might expect, agent sprawl compounds the problem. As agent populations grow across enterprise workflows, the governance gap multiplies.

From my view, here’s what needs to happen in an industry that has spent most of its governance energy on who can access what. The next layer is controlling what context agents reason against, and making that context canonical, verified, and auditable before the agent ever acts on it. Without that, you’re not really governing the agent; you’re governing the credential it runs under.

Agents will become increasingly capable and numerous, and the number of decisions they make will grow exponentially. But what makes a decision trustworthy hasn’t changed: You need to know what information shaped it, whether that information was accurate, and whether you can reconstruct the reasoning after the fact.

Ultimately, those are data management problems as much as they’re AI problems. And since AI is powered by data, it’s time for AI governance frameworks to catch up to that reality.