An LLM gateway is the control point between applications and model providers. It routes model requests, holds provider keys, enforces token budgets and filters prompts and responses. An MCP gateway is the control point between AI clients and MCP servers. It decides which tools each user can call, holds tool credentials and records each call. Agents in production usually need both.
The model side is already multi-provider. 37% of enterprises in a16z’s 2025 CIO survey ran five or more models in production, up from 29% a year earlier (a16z, June 2025). Switching is rarer than upgrading, though: only 11% of teams changed model providers in the past year, while 66% upgraded within their vendor (Menlo Ventures, July 2025).
This guide is for architects and platform leads deciding which layer to buy first. It compares what each gateway controls, traces one agent request through both and lists products that claim both. For the MCP layer on its own, start with what an MCP gateway does and how to evaluate one.
Last verified: October 25, 2026, against the vendor documentation linked below and MCP spec revision 2026-07-28.
Key Takeaways
- An LLM gateway governs model access: routing, provider keys, token budgets and prompt filters. An MCP gateway governs tool access: discovery, per-tool permissions, tool credentials and call records.
- MCP sampling is deprecated in the 2026-07-28 spec, so MCP servers that need a model will call providers directly. Put that traffic behind the LLM gateway too.
- Several products now document both layers, including Azure API Management and Amazon Bedrock AgentCore Gateway. Test whether one identity, policy and trace really cover both.
- Buy for the traffic you have: model calls first, tool calls when agents reach company systems.
Disclosure: ASCENDING builds Jarvis Registry, an open source MCP registry and gateway. Its public pages don’t describe an application-facing LLM gateway, and the comparison below is based on public documentation.
What does an LLM gateway control?
An LLM gateway controls how applications reach models. Applications send one request format to the gateway, and the gateway decides which provider and deployment serve it. That puts these controls in one place:
- Routing and fallback. The gateway spreads load across deployments and fails over when a provider has a problem. LiteLLM documents load balancing, fallbacks and retries across 100+ providers.
- Provider keys. Applications get a gateway key instead of each provider’s key, so provider secrets stay in one store.
- Token budgets and rate limits. Azure API Management documents token-per-minute limits and token quotas per consumer. Portkey documents limits based on cost or tokens.
- Caching. Semantic caches reuse answers to similar prompts to save tokens and time. Azure API Management and Portkey both document one.
- Prompt and response guardrails. Filters can redact personal data or block content before it leaves your boundary. Kong AI Gateway documents prompt guards and a sanitizer that redacts personal data.
- Usage logs. Cloudflare AI Gateway reports request, token and cost metrics.
If your main question is cost by team, see AI token cost tracking by users, teams and budgets. If it’s changing an application’s model, see switching AI models without rebuilding apps.
A gateway matters more as usage spreads. Anthropic held 32% of enterprise LLM API usage in mid-2025, OpenAI 25% and Google 20% (Menlo Ventures, July 2025). Each provider means another key, another bill and another log format without a gateway in front.
Source: Menlo Ventures survey of 150+ technical decision-makers, June 30 to July 10, 2025. Usage share is weighted by application scale.
What does an MCP gateway control?
An MCP gateway controls which tools agents can reach and what happens when they call one. A tool call is an action, such as reading a ticket or updating a record, so the risk sits in permissions and credentials rather than tokens. Many servers have no control at all: 40.55% of 7,973 live remote MCP servers exposed tools without authentication in a May 2026 measurement study (Zhou et al., arXiv).
- Discovery. The gateway shows each user only the tools they may use. Cloudflare’s MCP server portals list a server only for users who match an Allow policy.
- Per-tool authorization. Kong’s AI MCP Proxy evaluates deny lists before allow lists at default and per-tool levels. It’s an AI Gateway Enterprise feature that needs Kong Gateway 3.12 or later.
- Tool credentials. Amazon Bedrock AgentCore Gateway handles OAuth flows, token refresh and credential injection for each tool.
- Protocol bridging. The same AWS service converts APIs and Lambda functions into MCP tools.
- Call records. Kong writes every allowed and denied access attempt to an audit log. Portkey’s MCP gateway records each tool call with user identity, parameters and response.
Tool access also depends on how users sign in to the gateway. For the identity setup, see MCP OAuth with Entra ID or Okta.
MCP gateway vs LLM gateway, side by side
The two layers can share a vendor, but they inspect different traffic and hold different secrets.
| Row | LLM gateway | MCP gateway |
|---|---|---|
| Traffic it sees | Prompts and completions sent to model providers | Tool discovery and tool calls sent to MCP servers |
| Typical caller | Applications, agents and MCP servers that need a model | AI clients and agents acting for a user |
| Deployment model | SaaS, self-hosted proxy or part of a cloud API gateway | SaaS, self-hosted gateway or part of a cloud API gateway |
| Identity and access control | Gateway keys or tokens per app, team or user; model allow lists | User sign-in; per-server and per-tool permissions |
| Credentials it holds | Model provider API keys | Tool and SaaS credentials, often per user |
| Audit logging | Requests, tokens, cost, latency, sometimes prompts | Who called which tool, with which result |
| Supported MCP transports | None; it speaks provider HTTP APIs | Streamable HTTP; some also SSE or stdio (LiteLLM documents all three) |
| Hosting | Near applications and model endpoints | Near tool servers and company systems |
| Cost controls | Token budgets, quotas and caching | Rate limits on calls; tool costs live in downstream systems |
| Content guardrails | Prompt and response filters | Argument checks and tool allow lists |
| If it fails | Model calls stop or fall back | Agents lose tools; actions stop |
How does one agent request pass through both gateways?
One agent request usually crosses both gateways more than once. Here’s a support agent asked to update a ticket:
- The agent asks a model what to do. The request goes through the LLM gateway, which picks the provider, checks the budget and filters the prompt.
- The model asks for a tool. The agent sends that tool call to the MCP gateway, not to the model provider.
- The MCP gateway checks the user. It confirms the user may call this tool, injects the tool credential and records the call.
- The result goes back to the model. The next model call carries the tool output, so it passes through the LLM gateway again.
- The agent answers. One user request has produced at least two model calls and one tool call.
Two details decide whether you can investigate that request later. Both gateways should see the same user identity, not a shared service account. And both should emit spans into the same trace, so one ID links the model calls to the tool call.
There’s a newer wrinkle. MCP once let servers ask the client’s model for a completion, a feature called sampling. The 2026-07-28 spec deprecates sampling and says existing servers should integrate directly with LLM provider APIs. So an MCP server that summarizes or classifies data will call a model itself. Route those calls through the LLM gateway too, or they escape your budgets and filters.
Which products cover both layers?
Several products now document more than one kind of traffic. We reviewed each vendor’s own documentation on October 25, 2026 and recorded which traffic types it describes.
| Product | Model traffic | MCP traffic | Agent-to-agent (A2A) |
|---|---|---|---|
| Azure API Management AI gateway | Yes | Yes: expose REST APIs as MCP servers or pass through existing ones | Yes: import A2A agent APIs |
| Amazon Bedrock AgentCore Gateway | Yes: model-based routing across providers | Yes | Yes: passthrough targets |
| LiteLLM | Yes | Yes: Streamable HTTP, SSE and stdio | Yes |
| Portkey | Yes | Yes: documented as its MCP Gateway | Not described on the pages reviewed |
| Kong AI Gateway | Yes | Yes: AI MCP Proxy, Enterprise license | Mentioned |
| TrueFoundry AI Gateway | Yes | Yes | Not described on the page reviewed |
| Cloudflare AI Gateway | Yes | Not on the AI Gateway page; MCP server portals are a separate Cloudflare One feature | Not described |
Methodology. We read each product’s main gateway page and its MCP page where one exists. We recorded only what the vendor’s own pages state. We didn’t test the products, and a “Yes” says nothing about depth, licensing or maturity.
The trend is clear even in this small sample: the labels are converging. AWS now describes AgentCore Gateway as a single entry point for tools, agents and models. That doesn’t make the controls the same. One product can still keep separate keys, policies and logs for model traffic and tool traffic. When a product claims both, ask it to show one user, one policy change and one trace across a model call and a tool call.
Where does an agent gateway fit?
An agent gateway governs calls from one agent to another, usually over the A2A protocol. It’s a third kind of traffic: the caller delegates a whole task, not a single model call or tool call. Azure API Management and AgentCore Gateway both list A2A alongside models and tools. For how agent gateways differ from the API gateway you already run, see agent gateway vs API gateway.
Do you need an LLM gateway, an MCP gateway or both?
Buy for the traffic you actually have today, and plan the second layer before agents reach company systems.
| Your situation | What to add |
|---|---|
| Applications call models; no agents use company tools yet | An LLM gateway |
| Agents use a few tools through one client, such as an IDE | An MCP gateway, with identity from your IdP |
| Agents call models and company tools in production | Both, connected by one identity and one trace |
| MCP servers call models themselves | Route those calls through the LLM gateway as well |
| Agents delegate work to other agents | Add agent gateway controls for A2A traffic |
Before choosing one product for both layers, ask for five things in a demo:
- The same user identity on a model call and a tool call, not a shared service key.
- One policy change, such as removing a user from a group, that takes effect on both.
- One trace ID that links the model calls to the tool call.
- Where each credential lives: provider keys, tool credentials and per-user grants.
- Which features need a separate license or product.
The MCP gateway buyer’s guide covers hosting, tenancy and evidence checks for the MCP side.
Where does Jarvis Registry fit?
ASCENDING publishes this blog and maintains Jarvis Registry, an open source MCP registry and gateway. The capabilities below come from public product pages checked on October 25, 2026. They still need testing in your environment.
The Jarvis Registry page describes an MCP gateway for tools and an agent gateway for agent-to-agent calls. It lists RBAC plus a per-tool access list evaluated on every invocation, OpenID Connect and SAML 2.0 sign-in, and one OpenTelemetry span per tool call and agent hop. That page doesn’t describe model routing, provider keys or token budgets, so Jarvis Registry belongs on the MCP side of the table above.
Jarvis Chat is a separate employee chat workspace. Its page lists OpenAI, Anthropic and AWS Bedrock behind one interface, with access rules that scope models to users or teams. That’s a governed workspace, not an application-facing LLM gateway. If your applications need one, pair Jarvis with an LLM gateway and test that identity and trace IDs carry across both.
Bring one agent workflow that calls a model and a tool to a Jarvis demo, and run the five checks above against it. The Jarvis MCP gateway page lists the protocol standards it supports.
References
- a16z: How 100 Enterprise CIOs Are Building and Buying Gen AI in 2025
- Menlo Ventures: 2025 Mid-Year LLM Market Update
- Microsoft: AI gateway capabilities in Azure API Management
- AWS: Amazon Bedrock AgentCore Gateway
- LiteLLM: AI Gateway
- LiteLLM: MCP overview
- Portkey: AI Gateway
- Portkey: MCP Gateway
- Kong: AI Gateway
- Kong: AI MCP Proxy plugin
- TrueFoundry: AI Gateway
- Cloudflare: AI Gateway
- Cloudflare: MCP server portals
- MCP specification 2026-07-28: Sampling (deprecated)
LLM gateway and MCP gateway questions
What is the difference between an MCP gateway and an LLM gateway?
An LLM gateway sits between applications and model providers. It routes model requests, holds provider keys, enforces token budgets and filters prompts and responses. An MCP gateway sits between AI clients and MCP servers. It decides which tools each user can discover and call, holds tool credentials and records each tool call. One governs model access; the other governs tool access.
Do I need both an LLM gateway and an MCP gateway?
You need both once agents use company tools in production. Applications that only call models need an LLM gateway. Agents that reach company systems through MCP need an MCP gateway. When both exist, check that the same user identity and one trace ID carry across them so a single request can be investigated end to end.
Can one product be both an LLM gateway and an MCP gateway?
Yes. On October 25, 2026, documentation from Azure API Management, Amazon Bedrock AgentCore Gateway, LiteLLM, Portkey, Kong and TrueFoundry described both model and MCP traffic. One product doesn't guarantee one policy model, so test whether identity, policy rules, logs and licensing cover both kinds of traffic. See the Azure API Management AI gateway overview for one example.
Is an AI gateway the same as an LLM gateway?
Not always. Vendors use "AI gateway" both for model-only gateways and for products that also govern MCP tools and agent-to-agent traffic. Azure API Management's AI gateway, for example, covers language model APIs, MCP servers and A2A agent APIs. Ask which traffic types a product governs rather than relying on the label.
