An agent is not just a chat window with better manners.

It can call a model five times, search an internal system, open a ticket, write to a database, and hand a task to another agent before anyone notices the token count climbing.

That is useful. It also means the hardest question is no longer “which model should we use?”

It is:

What is this agent allowed to reach, spend, and change?

An agent gateway is the layer that answers that question in one place.

It sits between agents and the things they call: model providers, MCP tools, internal APIs, and sometimes other agents. It can authenticate the caller, decide whether a request is allowed, route it, limit cost, and leave a trail for the person who has to explain what happened later.

Without it, every agent tends to bring its own API keys, its own logging, its own routing logic, and its own interpretation of “just call the tool.”

That is fine for a prototype. It gets awkward very quickly in production.
Diagram showing an agent gateway enforcing identity, policy, budgets, and audit across model, tool, and agent traffic

First, what problem is an agent gateway solving?

Agents create three different traffic paths:

  1. Agent to model

    The normal LLM request. This is where model choice, rate limits, fallbacks, and token cost matter.

  2. Agent to tool

    Usually MCP traffic, but also direct HTTP APIs, databases, queues, or internal services. This is where the agent can read or change something real.

  3. Agent to agent

    One agent delegates work to another. This is often called A2A traffic.

A normal LLM gateway usually handles the first path well. It routes requests, tracks tokens, chooses fallbacks, and can apply prompt controls.

An MCP gateway handles the second path. It decides which tools an agent can discover and invoke, and it may handle the credentials used upstream.

An agent gateway tries to govern all of those paths with one policy model.

That distinction matters because model calls are mostly about what the agent says. Tool calls are about what the agent does.

The gateway is not the agent brain

This gets confused a lot.

The agent framework decides how to plan, reason, retry, and call tools. The gateway does not replace that.

The gateway is closer to a bouncer with a ledger:

  • Who is this agent?
  • Which model can it use?
  • Which tools can it call?
  • Can it use the production version of that tool?
  • Can it spend another $50 on a loop?
  • Should this request require approval?
  • What happened before and after the tool ran?

If you have ever put an API gateway in front of services, the idea is familiar. The difference is that the caller is now a system that may create more calls on its own.

Why model routing alone stops being enough

Model routing is useful. It can choose a cheaper model for simple work, switch providers during an outage, and stop one team from consuming every available request.

But model routing does not answer this:

Can the support agent issue a refund? Can the coding agent run a production migration? Can the research agent read a customer export? Can the finance agent call the payroll tool?

Those are tool-boundary questions.

The most dangerous agent behavior usually starts after the model has finished generating text. The model suggests a tool call. Then a credential reaches a real system. The tool returns data or changes state. The agent sees the result and decides what to do next.

That is the path a serious gateway needs to control.

What a production agent gateway should actually do

There are six things worth evaluating in any agent gateway. Everything else is detail until these are clear.

Control The question to ask Why it matters
Protocol coverage Does it govern LLM, MCP, and A2A traffic? Agents can use all three in one task.
Identity Can it identify the agent and the human or service behind it? One shared master key is not governance.
Tool policy Can it allow, deny, scope, and inspect tool calls? Tools read and change real systems.
Spend limits Can it stop a loop at a hard budget? Reporting cost after the loop finishes is not a limit.
Observability Can you trace the complete chain across model and tool calls? Incidents rarely stay inside one request.
Deployment Can it run where your data and credentials are allowed to be? Security rules often decide this before features do.

The two questions that separate a real evaluation from a feature checklist are:

Can it block a risky tool call before execution?

Can it stop an agent from spending more after a budget is reached?

If the answer to either is “we can see it in the dashboard later,” that is observability, not control.

Seven agent gateways worth putting on a shortlist

The list below is not a universal ranking. “Best” depends on where you run, which protocols you need, and whether you want a managed service or a platform your team owns.

Product capabilities change quickly in this category. Treat this as a shortlist of agent gateways for a proof of concept, then verify the current documentation, licensing, and operational requirements before a long-term decision.

1. Bifrost

Bifrost

Best fit: Teams that want one self-hosted gateway governing LLM, MCP, and agent-to-agent (A2A) traffic, especially where VPC, on-prem, or air-gapped deployment matters.

Bifrost combines model routing and MCP access behind an OpenAI-compatible endpoint. Its published governance model centers on virtual keys, which can scope models, providers, tools, budgets, and rate limits per agent, team, or customer.

The useful part is not the provider count. It is being able to give a coding agent one identity with a limited tool bundle and a hard budget, instead of handing it a general-purpose key that works everywhere.

Its published MCP features include curated virtual MCP tool bundles, layered tool filtering, upstream OAuth support, and deployments inside a customer VPC or on-premises environment. Teams evaluating latency-sensitive workloads can also review its published benchmark methodology and results.

For agent-to-agent traffic, Bifrost has a documented Agent Gateway with A2A support. You register an existing agent from its agent card, control which callers can reach it, forward credentials, handle push callbacks, and inspect each run in Agent Logs. That puts LLM, MCP, and agent-to-agent traffic behind one gateway.

Ask during evaluation: Can the same policy identity govern the model request, the MCP tool call, and the A2A agent call?

2. Kong AI Gateway

Best fit: Organizations already running Kong Gateway or Kong Konnect for API management.

Kong AI Gateway is compelling when the infrastructure team already trusts Kong with normal API traffic. Instead of introducing a separate AI control plane, the team can extend the existing one to LLM, MCP, and A2A traffic.

Kong publishes central authentication, authorization, rate limiting, token quotas, audit trails, model routing, prompt controls, and MCP server generation on top of managed APIs.

The tradeoff is obvious too. If you do not already use Kong, adopting a broad API management platform only for agent traffic may be more platform than you need.

Ask during evaluation: Are tool-level controls as mature as model-side controls for the exact MCP flows you run?

3. LiteLLM

Best fit: Teams that want an open source, self-hosted Python proxy covering LLM routing, MCP tools, and A2A agents.

LiteLLM started as a way to call many model providers through one interface, and its proxy now includes an MCP gateway and an Agent Gateway for A2A agents. It is MIT licensed and runs in your own environment.

On the A2A side, you register agents in the admin UI or in config.yaml, and the proxy serves each agent's card and forwards calls. Access is checked per virtual key and team, and the logs show who called which agent along with latency and cost. Its docs also cover per-agent iteration budgets and a kill switch. A2A agents can sit behind the proxy next to model and MCP traffic, which keeps the setup in one place.

It is a Python service you operate yourself, so plan for the database, upgrades, and scaling. Some governance features, such as SSO and audit logs, are listed under its enterprise offering, so check what the open source tier covers for your requirements.

Ask during evaluation: Can your team own the proxy, its database, and its upgrade path, and which governance features does your tier include?

4. Amazon Bedrock AgentCore Gateway

Best fit: AWS-native teams that want a fully managed gateway and are comfortable keeping the control plane inside AWS.

Amazon Bedrock AgentCore Gateway can turn APIs, Lambda functions, and existing services into MCP-compatible tools. It supports inbound identity checks, outbound credential injection, OAuth flows, tool search, model routing, and passthrough targets for agent-to-agent traffic.

This is the quickest route for a team already building on Bedrock and AWS identity. There is less gateway infrastructure to run and fewer components to assemble.

The limit is also clear. It is a managed AWS service. If you need on-prem deployment, a multi-cloud control plane, or an air-gapped environment, this is probably not the right baseline.

Ask during evaluation: Which credentials remain AWS-managed, and which identities can be passed through to downstream tools?

5. IBM ContextForge

Best fit: Teams that need to federate a messy real-world collection of MCP servers, REST APIs, gRPC services, and agents.

IBM ContextForge is more about discovery and federation than it is about being only a model router. It can put multiple internal services behind one endpoint, translate REST or gRPC services into tool-friendly forms, and connect external agents through A2A.

That makes it interesting when the core problem is not “which model do I route to?” but “how do I give agents a safe catalog of twenty years of internal APIs?”

Its published capabilities include user-scoped OAuth, retries, rate limits, OpenTelemetry, an admin UI, and air-gapped deployment support.

Ask during evaluation: Do you need per-agent hard budget enforcement, or is tool and service federation the primary problem?

6. Agent Router

Best fit: Teams already committed to Envoy, Kubernetes Gateway API, and a platform-owned data plane.

Agent Router is the project previously known as Envoy AI Gateway. It is built as an additive agent and model layer around Envoy Gateway.

It publishes model-provider routing, token-aware quotas, MCP multiplexing, tool filtering, OAuth with PKCE, CEL-based authorization, and OpenTelemetry metrics and traces. It also has a local mode for trying the gateway without a full Kubernetes environment.

The project is a strong fit for teams that want an Envoy-shaped control plane. The open-source path leaves several platform concerns to you, including operational UI, SSO, RBAC, and some guardrail choices.

Ask during evaluation: Are you buying an Envoy foundation or trying to avoid building one?

7. Azure API Management

Best fit: Teams already running Azure API Management, or building on Azure, that want AI traffic governed as part of a larger managed API platform.

Microsoft's AI gateway is not a separate product. It is a set of capabilities on Azure API Management, so the policies, monitoring, and developer portal your API team already uses extend to AI traffic.

For models, it offers an LLM token limit policy (tokens per minute or periodic quotas, keyed by subscription, IP, or a custom expression), backend load balancing, and circuit breakers. For tools, it can expose a REST API or an existing server as an MCP server, with policies for rate limits, JWT validation, and IP filtering. For agents, it can import A2A agent APIs, expose the agent card through the gateway, and apply policies to the traffic. Monitoring runs through Azure Monitor and Application Insights.

Check the details against your plan. Microsoft's docs label the unified model API and the AI gateway integration in Microsoft Foundry as preview. The MCP and A2A pages I reviewed carry no preview label, but they list limits: MCP support covers tools but not resources or prompts, MCP policies currently apply to all tools on a server, and A2A import supports JSON-RPC agents only. Some newer AI gateway features arrive first through an early release channel on classic tiers.

Ask during evaluation: Which of the features you need are generally available on your API Management tier today, and which are still in preview?

The shortcut comparison

If you need... Start with...
One self-hosted gateway for LLM, MCP, and A2A traffic with VPC or on-prem options Bifrost
Existing Kong operations and broad API governance Kong AI Gateway
An open source, self-hosted Python proxy for LLM, MCP, and A2A LiteLLM
Fully managed AWS integration Bedrock AgentCore Gateway
Federation of MCP, REST, gRPC, and agent services IBM ContextForge
Envoy and Kubernetes Gateway API as the data plane Agent Router
A managed gateway inside an Azure and API Management footprint Azure API Management

That is a starting point, not a purchase decision.

The tool boundary is the real test

Every vendor can show you a routing demo.

The better test is a tool call that should be denied.

For example:

Agent: “Delete the inactive customer records.” Tool request: DELETE /customers?status=inactive Policy: this agent may read customer data, but may not delete it. Expected result: tool execution never happens.

Now make it harder:

Agent: “Issue a refund under $50.” Tool request: refund(customer_id, amount) Policy: agent may refund up to $50 only for the current user’s account. Expected result: identity, amount, ownership, and tool argument rules all pass before execution.

Then test the return path:

Tool result: contains an access token or another customer's data. Expected result: the result is blocked or redacted before the model sees it.

That last part gets missed. A tool can be allowed to run and still return something the model should not receive.

You want controls before execution and controls after execution.

An agent gateway evaluation that will not waste three weeks

Do not start with a vendor demo. Start with three real agent journeys from your environment.

Journey 1: the normal safe path

An agent reads an internal document, summarizes it with an approved model, and opens a draft ticket. This proves ordinary routing, identity, audit, and policy work.

Journey 2: the request that must stop

An agent tries a tool call outside its scope. This proves the gateway can prevent execution rather than simply report it.

Journey 3: the runaway loop

An agent is intentionally given a task that retries, switches models, and makes repeated tool calls. This proves budgets and rate limits are enforced before the invoice arrives.

For every journey, ask for four pieces of evidence:

  1. The caller identity
  2. The policy decision
  3. The upstream request and result, with sensitive data redacted
  4. The trace that connects the model call, tool call, and final agent output

If a product cannot make that evidence easy to collect, incident response will be unpleasant later.

The questions people forget to ask

Can it use the end user’s identity?

An agent acting as the user should not quietly become a service account with broader access. Check whether the gateway can exchange, forward, or scope identity per user and per tool.

Can it kill access instantly?

If a tool server is compromised or an agent starts behaving badly, can you disable one agent key or tool bundle immediately without redeploying every agent?

Are budgets hard limits?

Cost dashboards are useful. A budget that only alerts after a request succeeds is not a budget. Test whether the next request is rejected once the limit is reached.

What happens during provider failure?

Fallbacks can improve reliability. They can also accidentally send a sensitive prompt to a provider you did not approve. Make provider fallback policy explicit.

Does it only cover the happy transport?

MCP has multiple transports. Tools may use HTTP, SSE, streamable HTTP, or stdio. Ask which path gets the same identity, policy, audit, and guardrail coverage.

The honest conclusion

An agent gateway is not a magic layer that makes autonomous systems safe.

It is the place where you can finally make the rules visible and enforceable.

The right agent gateway is usually not the one with the longest provider list. It is the one that can live where you need it to live, intercept the traffic you actually have, stop the tool call you are afraid of, and leave enough evidence behind for someone else to understand the decision.

Start with your risky tool call. Everything else gets clearer after that.

Sources