In our previous article, we talked about AI governance moving "down the stack": i.e. policy has to become a machine-enforceable decision made in the path of each request to AI models.

A model gateway is hence the obvious place to put in that path.

It is also, quite often, unfortunately where the thinking stops. I hope this article keeps the cogs whirring.

For many enterprises, an AI gateway is definitely a most sensible first move, but the problem starts when the architecture diagram labels it “AI control plane” and then everyone goes home.

A model gateway can carry and enforce controls on AI traffic, but that doesn’t make it the enterprise’s control plane.

Example model gateways

There are three broad groups of products you will encounter in this category:

  1. Open source: LiteLLM, Envoy AI Gateway, agentgateway;
  2. API gateways with added AI features: Kong, Azure API Management, Apigee;
  3. Managed services: Cloudflare AI Gateway, Portkey.

A newer category, MCP gateways (Docker, Composio, et al.), handle tool calls rather than model calls.

What gateways are good at

To be fair to gateways, (and they do certainly deserve it), a good one gets you a great deal of bang for the buck:

  • A single front door for model traffic, instead of API keys scattered across various repositories;
  • Provider abstraction, so applications aren’t hard-wired to one particular vendor’s SDK;
  • Authentication of calling applications;
  • Rate limits, quotas and cost attribution;
  • Model allow-lists, routing and failover;
  • Content filtering, redaction and request logging.

Some products even go considerably further: with per-user policy, tool proxying and integration with external policy engines.

If you don’t have a gateway yet, you probably do need one.

Two planes, two jobs

Infrastructure people have separated these two responsibilities for years.

  • Data plane: handles the requests and responses themselves. For AI, that means mediating traffic between applications, models and, where applicable, tools.
  • Control plane: defines and coordinates the identities, policies, configuration and governance decisions that determine what the data plane should permit, and how those decisions are evidenced.

Service meshes make this more concrete:

  • Istio manages traffic between the many small services that make up a modern application. A lightweight proxy called Envoy sits beside each service and handles its traffic. That’s the data plane. A separate control plane tells every proxy what the rules are.[1]
  • NIST’s Zero Trust architecture, the US government’s reference model for checking every access request, names the parts explicitly. A policy decision point decides whether access should be granted. A policy enforcement point, sitting in the path of the request, applies that decision.[2]
  • Open Policy Agent (OPA) is a widely used open-source tool built around this same split. A service asks OPA whether an action is allowed, OPA checks its policy and then answers, with the service enforcing the answer.[3]

The important word in all of the above is logically. One product can implement both planes, and plenty do. The distinction isn’t about how many boxes are on the diagram; it’s about which questions each part is answering.

“AI control plane” isn’t a standardised product category yet, and I’m not going to pretend the industry has agreed on one. That's part of what this series of articles is about. I mean it here as a set of responsibilities.

A gateway naturally answers questions about the traffic in front of it. The control plane has to answer a much wider set:

  1. Who or what is making the request?
  2. On whose behalf is it acting?
  3. Which resource or operation is being requested?
  4. Which version of policy governs that request?
  5. Where is the authorisation decision made, and where is it enforced?
  6. What evidence records the decision and the resulting execution?

A gateway can contribute to every one of the above. The snag is that several of them concern data that may never pass through it.

One agent, three operations

Let's take as an example Sam, a developer, asking an AI coding assistant to fix a failing test. The assistant is a coding agent, software that can take actions as well as answer questions. It has its own login and access to a few different tools.

  1. The agent asks a model for help. The request goes through the gateway, which checks the agent’s credentials, confirms the model is approved and the team has budget, and passes the request on. The answer comes back; everything worked as designed.
  2. It looks up code. To understand the problem, the agent searches the company’s code. The search service logs in with a shared account that can read every repository in the organisation, because that was the easiest way to make search useful. Some of the results come from a project Sam himself isn’t allowed to see.
  3. It changes the code. The model suggests a fix. The model doesn’t make the change itself; the agent does, using a GitHub tool that holds a key with write access to the organisation’s repositories. The change goes in. It happens to include an edit to the automated build pipeline.

What did the gateway check?

It checked that an approved application called an approved model, within budget, and got an answer. Some gateways also look at the answer and can see that the model suggested a change.

On its own, it didn’t check:

  • Whether the agent was acting for Sam, and what he’d allowed it to do;
  • Whether Sam was allowed to see the code that came back;
  • Whether Sam was allowed to change that repository, let alone its build pipeline;
  • Which rules applied to each step, and which version of them;
  • How the three steps link together afterwards.

The search and the change went to different systems, using different keys, and may never even have passed through the gateway at all. Some gateways handle tool traffic too. Even then, they can only make a good decision if they know who the agent is acting for and what it’s been allowed to do.

Somebody has to own that information, and that’s the control plane’s job. The guard waved through the request at his door; this doesn’t mean everything after it was authorised.

A key isn’t permission

Look again at step three, where the agent is changing the code:

The agent used a digital key that someone maybe set up months ago to get the GitHub integration working. To save time, they gave it write access to every repository in the organisation.

GitHub has done its job just fine. It checked the key and saw that the key was allowed to make the change. But the key only shows what the integration can do. It says nothing about what Sam asked the agent to do on this particular day.

Think of a master key you've had cut for a contractor. Every door accepts it, but that doesn’t mean the contractor was asked to go into every room.

What was missing is the question nobody asked: is this agent, working for Sam, allowed to make this particular change, to this repository, right now?

OWASP lists this in its top ten risks for AI applications. It calls it Excessive Agency: AI systems given more access or freedom than the task actually needs, including tools that can reach data the person using them couldn’t see themselves.[4] That’s not an AI problem. It’s an old permissions problem in a new setting.

The standards are however starting to catch up. e.g. MCP is of course a widely adopted standard for connecting AI agents to tools.

MCP’s HTTP authorisation model makes the boundary explicit: an MCP server must validate tokens intended for that server, and must not simply pass a client’s token through to a downstream service.[5] i.e. The checks happen at each door, which is exactly where a single gateway can’t be.

Where should the rules live?

Here’s the argument worth having.

The rules for what AI is allowed to do should be written once, in one place the organisation controls, and checked at every door. The gateway is one of those doors. So are the code search service, the GitHub tool and the agent itself.

Policy can be authored centrally and evaluated locally, provided each enforcement point can identify the policy version it used with the organisation defining how updates, stale policies and failures are handled.

The gateway should apply the rules; it shouldn't quietly become the place where they're written.

Now, experienced engineers may push back, and three objections deserve a fair hearing:

  • "Just send everything through the gateway." You can try. But agents reach tools in many different ways, across different systems and suppliers. One running on a developer's laptop or in an automated build may never pass the gateway at all. A checkpoint that people can walk around by accident is more of a suggestion than a proper checkpoint.
  • "Checking a central rulebook every time will slow things down." In fact, it needn't. Each door can keep its own copy of the rulebook and check locally, as long as every copy is the same, up-to-date version. This is how tools like OPA are commonly run.[3]
  • "This is overkill for us." Possibly. A very small organisation with one AI application and one supplier might sensibly just keep everything in one place. That's fine. The jobs still exist; you've simply decided they all sit together. Just write that decision down.

In practice, the opposite problem is far more common: Each door slowly picks up its own rules: some in the gateway's settings, some in what each key can do, some in the search service's access lists, some in a configuration file nobody remembers writing. Nobody designed it that way. It just ended up growing out of all proportion.

At this point you don't have an AI control plane. You just have a bunch of opinions.

Evidence that outlives the hop

The gateway keeps a log, which is useful; but it only records what happened at its own door.

To work out afterwards what happened with Sam’s failing test, you need to connect all three steps: asking the model, searching the code and changing it. For each step, you’d want to know:

  • who asked for it (Sam);
  • who did it (the agent);
  • which key was used;
  • which rules applied, and which version of them;
  • what was decided, and what happened.

The records don’t even all have to sit in the same place. Giving every step in the task the same reference number, much like a case number, is enough to link them together later.

N.B. That record also has to outlast your tools. If your only history sits inside the gateway, then swapping gateways, AI suppliers or agent software means losing the record with it. Tools should be replaceable; the evidence shouldn’t end up leaving with them.

One product is fine. Missing owners aren’t.

To be crystal clear about what I’m not saying: gateways CAN apply rules and you certainly don’t necessarily need a separate product for the control plane, and no single design suits every organisation.

What I am saying is that each of those six questions from earlier needs someone, or something, clearly responsible for answering it.

“We have a gateway” doesn’t answer the hard ones on its own.

The jobs split three ways:

  • The control plane decides who’s who and what the rules are.
  • Every door, the gateway included, applies those rules.
  • The evidence lets you piece together what happened at every step.

In Sam’s case, the request to the model was handled perfectly. The code search and the change went ahead because the keys they used happened to allow it. You need to govern the whole task, not just the conversation with the model.

A gateway can apply the rules at one door. A control plane makes sure every door applies the same rules, and keeps a record of how they were applied.

Which raises the question for the next article in this series:

if a gateway’s logs aren’t the governance record, what is?

*Jason Southgate leads Parakletos AI, a London-based AI and cloud consultancy focused on production AI systems where governance, reliability, and accountability matter.*

He has completed a University of Cambridge programme in data science, machine learning and AI.

Parakletos AI helps organisations use AI without losing control. They put identity, permissions and an audit trail around AI models and tools, so teams can move quickly and the business can account for what happened. This makes the governed route easier than the unmanaged one.

If you would prefer to buy a product rather than build, they can also help you.

References

[1] Istio, Architecture. Logical separation of an Istio service mesh into a data plane of Envoy proxies and a control plane that manages and configures them.

[2] NIST SP 800-207, Zero Trust Architecture, 2020. The policy decision point (policy engine and policy administrator) and its separation from the policy enforcement point.

[3] Open Policy Agent, Introduction. Decoupling policy decision-making from policy enforcement, with policy evaluated locally by the service that needs it.

[4] OWASP, LLM06:2025 Excessive Agency. Excessive functionality, permissions or autonomy in LLM-based systems, including tools acting with privileged identities.

[5] Model Context Protocol, Authorization specification, 2025. Optional HTTP authorisation model: MCP servers as OAuth resource servers, audience-bound tokens and the prohibition on token passthrough.