Post 1 of 8 in the GenAI gateway series.

The first agent is easy. You pick a model, give the agent credentials, connect its tools, and test the result. You know which model it calls and where to look when something goes wrong.

Then another team builds an agent on a different model. A third team joins. Each agent works, but a month later someone forwards the model bill and asks a short question nobody can answer: which of these agents spent this?

Other questions follow. Which agents can use the most expensive model? What data can they send? Who checks that the organisation's rules apply to every team?

Customers ask me some version of this all the time. Before this series, I deployed a governed agentic platform from an empty AWS account, and one piece of it kept coming up in every conversation: the layer that every model call passes through. So I spent the last two weeks deploying a gateway privately on AWS, testing each feature, and keeping notes on what broke. This series is those notes.

It starts with questions, because they explain why a gateway earns a place in an agent platform. Try answering the ones below for your own setup.

The setup that creates the questions

Picture ten agents across four teams: a support agent, an incident investigator, a code reviewer, a release checker and an inventory analyst. Between them they call five models, chosen for different tasks and costs.

Each agent talks to its model directly. Each team wrote its own client code, picked its own retry settings, and decided for itself what counts as a safe prompt.

One task is rarely one model call. An incident investigator asks a model what to check, calls a tool, sends the result back, and decides whether to dig further. A retry or a loop adds more calls. Whatever rules you have must cover the whole sequence.

Every team's choices make sense on their own. Look across all ten agents and new jobs appear that no single team owns: access, spending, data policy, reliability and visibility.

I hit this early in my own lab. I had one test team in the gateway. When I added a second team for a workload agent, I had to decide its budget, its rate limit and the two models it could call. The gateway gave me one place to configure those controls for both teams.

The questions

Who is allowed to call what?

  • Which agent can call which model? Is that written down anywhere, or is it whatever the credentials happen to allow?
  • When a new agent needs a model, what do you hand it? If the answer is cloud credentials, how narrow are they?
  • One agent's credentials leak. Can you revoke that agent alone, without breaking the other nine?
  • Does an experimental agent have the same model access as your production support agent?

Access defined in ten places is understood in none.

Who spent the money?

  • Suppose the bill went up 40% last month. Which agent, which team, which model caused it?
  • An agent gets stuck in a loop at 2 a.m. and keeps calling your largest model. What stops it?
  • Can you give one team a monthly budget and have it enforced, not just reported after the fact?
  • Do you know the per-token price of every model you use? Who updates it when prices change?

A budget you only read at month end is a receipt, not a limit.

Where are the organisation's data and content rules applied?

  • A tool returns output with instructions that try to redirect the agent. What checks it before the next model call?
  • A tool returns customer details or an access key. Can those values reach the model, and which policy decides?
  • Do all ten agents apply the same content policy? Who would notice if one didn't?
  • When the policy changes, how many codebases need a release?

If you can't verify that a check ran, you can't rely on it.

What happens when a model has a bad day?

  • A model starts throttling. Does each agent retry, fail, or hang?
  • Is there a backup model? Has anyone tested whether the agent still completes its task with it?
  • A model version is retired. How many agents break, and how do you find them?

A fallback that returns an answer your agent can't use isn't a fallback.

Can you see what's happening?

  • Can you trace yesterday's model calls to an agent, a team and a task, with latency, token usage and estimated cost?
  • An agent made a poor decision last Tuesday. Can you find the calls behind it, within what your data policy lets you keep?
  • A task took too long. How much of that was model calls, and where do you look for the rest?

Model logs that don't line up with the agent's own trace give you pieces of a task, not the task.

How fast can you change models?

  • A cheaper model ships that's good enough for half your agents. How many code changes does it take to move them?
  • Someone wants to try an open-weight model for a week. How long before they can?
  • By the end of my lab, my gateway listed more than 80 models. Picture granting ten agents access to that list one codebase at a time.

A shared model name turns a code change into a config change. It doesn't remove the need to test each agent on the new model.

Are you paying for the same answer twice?

  • How many of today's model requests repeat an earlier one, and which responses are safe to reuse?
  • Your agents send the same long system prompt with every call. Are you paying full price for it every time?

A shared view makes it easier to spot repeated requests across agents. And a cached answer is only safe if it's right for the agent asking and the data hasn't changed.

The shape of every answer

Every question here is a decision that spans teams: who can use a model, what they can send, how much they can spend, and what happens when a call fails. Leave each decision inside each agent and you maintain ten versions of the same control.

That is why agents need a gateway: one service between them and every model they call.

Configure it, and the gateway becomes the one place where model access is checked, usage is attributed, budgets and rate limits apply, inputs and outputs are screened, and retries and fallbacks follow one policy. Agents call one interface, and every call is recorded the same way.

Teams still own their agents' reasoning, tools and results. The platform owner owns the controls on the model call.

What a gateway doesn't answer

A gateway only governs traffic that goes through it. Until agents lose direct access to models, it's a suggestion, not a control.

It also doesn't decide whether an agent may run a tool, change a record or act on an answer. A budget on model calls doesn't limit what the agent does between them. Screening a prompt doesn't guarantee the agent resists every injection attempt.

And it's a service to run. Someone has to secure it, monitor it and plan for it failing, because every agent depends on it. The next post covers that work.

What to take from this

  • Agent count isn't the test. The test is whether you can enforce your rules on model calls and show that you did.
  • One production agent can justify a gateway. The case gets stronger when several teams need shared controls over model access and spending.
  • List the access, spending and data rules that must hold across every agent. Those are the gateway's job. Give it an owner.

Your agents differ in almost everything except the model call. That's where the shared rules belong.

Next in the series: deploying the gateway on AWS.