An LLM gateway controls the model-request path: which provider an application calls, which key it uses and how usage is limited. An MCP gateway controls the tool-request path between an AI client and an MCP server. The exact permissions, budgets and logs depend on the product and setup.
Evaluate both paths before buying another gateway. Your current client, application or API platform may already provide the controls you need.
The model side is already multi-provider. In a16z’s 2025 CIO survey, 37% of respondents used five or more models in production, up from 29% a year earlier (a16z, June 2025).
Switching providers was less common in a separate survey: 11% of teams changed provider in the past year, while 66% upgraded within their vendor (Menlo Ventures, July 2025). These are dated survey findings, not a measure of your team’s control gaps.
This guide is for architects and platform leads deciding which layer to buy first. It compares what each gateway controls, traces one agent request through both and lists products that claim both. For the MCP layer on its own, start with what an MCP gateway does and how to evaluate one.
Last verified: October 25, 2026, against the vendor documentation linked below and MCP spec revision 2026-07-28.
Key Takeaways
- An LLM gateway governs model access; an MCP gateway governs tool access. Verify routing, permissions, credential handling and evidence in the selected product and connection mode.
- The 2026-07-28 spec deprecates MCP sampling but does not remove it immediately. Plan the recommended provider-API migration and include server-originated model calls in your control review.
- Several products now document both layers, including Azure API Management and Amazon Bedrock AgentCore Gateway. Test whether one identity, policy and trace really cover both.
- Buy for a control gap you can demonstrate. Existing client, service or API-platform controls may already cover a small workflow.
Disclosure: ASCENDING builds Jarvis Registry, an open source MCP registry and gateway. Its public pages don’t describe an application-facing LLM gateway, and the comparison below is based on public documentation.
What does an LLM gateway control?
An LLM gateway controls how applications reach models. Applications send one request format to the gateway, and the gateway decides which provider and deployment serve it. That puts these controls in one place:
- Routing and fallback. The gateway spreads load across deployments and fails over when a provider has a problem. LiteLLM documents load balancing, fallbacks and retries across 100+ providers.
- Provider keys. Applications get a gateway key instead of each provider’s key, so provider secrets stay in one store.
- Token budgets and rate limits. Azure API Management documents token-per-minute limits and token quotas per consumer. Portkey documents limits based on cost or tokens.
- Caching. Semantic caches reuse answers to similar prompts to save tokens and time. Azure API Management and Portkey both document one.
- Prompt and response guardrails. Filters can redact personal data or block content before it leaves your boundary. Kong AI Gateway documents prompt guards and a sanitizer that redacts personal data.
- Usage logs. Cloudflare AI Gateway reports request, token and cost metrics.
If your main question is cost by team, see AI token cost tracking by users, teams and budgets. If it’s changing an application’s model, see switching AI models without rebuilding apps.
A gateway matters more as usage spreads. Anthropic held 32% of enterprise LLM API usage in mid-2025, OpenAI 25% and Google 20% (Menlo Ventures, July 2025). Each provider means another key, another bill and another log format without a gateway in front.

Source: Menlo Ventures survey of 150+ technical decision-makers, June 30 to July 10, 2025. Usage share is weighted by application scale.
What does an MCP gateway control?
An MCP gateway controls which tools agents can reach and what happens when they call one. A tool call requests an operation, such as reading a ticket or updating a record. Check who may run that operation and which account reaches the business system.
Authentication remains a concern. A May 2026 study found that 40.55% of its 7,973 live remote MCP servers exposed tools without authentication (Zhou et al., arXiv). That finding describes the study’s sample; it does not establish the state of every server today.
- Discovery. The gateway shows each user only the tools they may use. Cloudflare’s MCP server portals list a server only for users who match an Allow policy.
- Per-tool authorization. Kong’s AI MCP Proxy evaluates deny lists before allow lists at default and per-tool levels. It’s an AI Gateway Enterprise feature that needs Kong Gateway 3.12 or later.
- Tool credentials. Amazon Bedrock AgentCore Gateway handles OAuth flows, token refresh and credential injection for each tool.
- Protocol bridging. The same AWS service converts APIs and Lambda functions into MCP tools.
- Call records. Kong writes every allowed and denied access attempt to an audit log. Portkey’s MCP gateway records each tool call with user identity, parameters and response.
Tool access also depends on how users sign in to the gateway. For the identity setup, see MCP OAuth with Entra ID or Okta.
MCP gateway vs LLM gateway, side by side
The two layers can share a vendor, but they inspect different traffic and hold different secrets. The rows below are evaluation areas, not a guarantee that every product implements every control.
| Row | LLM gateway | MCP gateway |
|---|---|---|
| Traffic it sees | Prompts and completions sent to model providers | Tool discovery and tool calls sent to MCP servers |
| Typical caller | Applications, agents and MCP servers that need a model | AI clients and agents acting for a user |
| Deployment model | SaaS, self-hosted proxy or part of a cloud API gateway | SaaS, self-hosted gateway or part of a cloud API gateway |
| Identity and access control | Gateway keys or tokens per app, team or user; model allow lists | User sign-in; per-server and per-tool permissions |
| Credentials it holds | Model provider API keys | Tool and SaaS credentials, often per user |
| Audit logging | Requests, tokens, cost, latency, sometimes prompts | Who called which tool, with which result |
| Supported MCP transports | None; it speaks provider HTTP APIs | Streamable HTTP; some also SSE or stdio (LiteLLM documents all three) |
| Hosting | Near applications and model endpoints | Near tool servers and company systems |
| Cost controls | Token budgets, quotas and caching | Rate limits on calls; tool costs live in downstream systems |
| Content guardrails | Prompt and response filters | Argument checks and tool allow lists |
| If it fails | Model calls stop or fall back | Agents lose tools; actions stop |
How does one agent request pass through both gateways?
In an architecture that uses both layers, one agent request can cross the gateways more than once. Here is an illustrative support-agent workflow:

Illustrative architecture. Correlated identity, authorization and trace records are requirements to demonstrate, not verified behavior of every gateway or deployment.
- The agent asks a model what to do. The request goes through the LLM gateway, which picks the provider, checks the budget and filters the prompt.
- The model asks for a tool. The agent sends that tool call to the MCP gateway, not to the model provider.
- The MCP gateway checks the user. It confirms the user may call this tool, injects the tool credential and records the call.
- The result goes back to the model. The next model call carries the tool output, so it passes through the LLM gateway again.
- The agent answers. One user request has produced at least two model calls and one tool call.
For an agent acting for an employee, retain enough evidence to connect that employee, the agent and the downstream action. A shared service key alone cannot identify the person behind the request. An autonomous workload instead needs its own accountable identity. In either case, test whether a request or trace ID links the model and tool records; do not assume that using one vendor correlates them automatically.
MCP sampling lets a server request a model completion through the client. The 2026-07-28 spec deprecates sampling but retains it for at least twelve months. It advises new implementations not to adopt it and existing implementations to migrate to provider APIs.
Deprecation does not change a running server’s route. Inventory any model calls the server makes itself, then decide where to control their keys, spend, data flow and logs.
Which products cover both layers?
Several products now document more than one kind of traffic. We reviewed each vendor’s own documentation on October 25, 2026 and recorded which traffic types it describes.
| Product | Model traffic | MCP traffic | Agent-to-agent (A2A) |
|---|---|---|---|
| Azure API Management AI gateway | Yes | Yes: expose REST APIs as MCP servers or pass through existing ones | Yes: import A2A agent APIs |
| Amazon Bedrock AgentCore Gateway | Yes: model-based routing across providers | Yes | Yes: passthrough targets |
| LiteLLM | Yes | Yes: Streamable HTTP, SSE and stdio | Yes |
| Portkey | Yes | Yes: documented as its MCP Gateway | Not described on the pages reviewed |
| Kong AI Gateway | Yes | Yes: AI MCP Proxy, Enterprise license | Mentioned |
| TrueFoundry AI Gateway | Yes | Yes | Not described on the page reviewed |
| Cloudflare AI Gateway | Yes | Not on the AI Gateway page; MCP server portals are a separate Cloudflare One feature | Not described |
Methodology. We read each product’s main gateway page and its MCP page where one exists. We recorded only what the vendor’s own pages state. We didn’t test the products, and a “Yes” says nothing about depth, licensing or maturity.
The trend is clear even in this small sample: the labels are converging. AWS now describes AgentCore Gateway as a single entry point for tools, agents and models. That doesn’t make the controls the same. One product can still keep separate keys, policies and logs for model traffic and tool traffic. When a product claims both, ask it to show one user, one policy change and one trace across a model call and a tool call.
Where does an agent gateway fit?
An agent gateway governs calls from one agent to another, usually over the A2A protocol. It’s a third kind of traffic: the caller delegates a whole task, not a single model call or tool call. Azure API Management and AgentCore Gateway both list A2A alongside models and tools. For how agent gateways differ from the API gateway you already run, see agent gateway vs API gateway.
Do you need an LLM gateway, an MCP gateway or both?
Start with the control gap in your actual traffic. Tool access and model access both need deliberate controls, but neither automatically requires another standalone gateway.
Find which path is failing
Consider a support assistant that answers a question and then looks up an invoice. The following are possible diagnostic cases, not reported test results:
| What you observe | Where to start | What to establish |
|---|---|---|
| The model provider rejects the request for exceeding a quota | Model-request path | Which provider or gateway limit was reached, and which application owns the usage |
| The model responds, but the invoice tool denies access | Tool-request path | Which server grant and downstream identity were used; changing model routing will not grant invoice access |
| Both requests succeed, but the answer uses the wrong invoice | Source and task evaluation | Whether the tool returned the right record and the model used it correctly; successful routing alone does not prove answer quality |
Keep the request IDs and observed error with the finding. Choose a new control only after identifying what the existing path cannot do.
Choose the control, then the product
| Your situation | Control decision to evaluate |
|---|---|
| Applications call models; no agents use company tools yet | Compare required budgets, routing and evidence with the model-provider and application controls already available |
| Agents use a few tools through one client, such as an IDE | Check whether client and server controls suffice; evaluate an MCP gateway when you need shared discovery, identity or resource policy |
| Agents call models and company tools in production | Evaluate controls on both paths; one platform or separate gateways may fit, with demonstrated identity and evidence correlation |
| MCP servers call models themselves | Include those model requests in the budget, credential and data-boundary design; route through an LLM gateway if that is the chosen control point |
| Agents delegate work to other agents | Assess A2A identity, authorization and task-routing requirements; verify what the existing platform supports |
Before choosing one product for both layers, ask for five things in a demo:
- The same user identity on a model call and a tool call, not a shared service key.
- One policy change, such as removing a user from a group, that takes effect on both.
- One trace ID that links the model calls to the tool call.
- Where each credential lives: provider keys, tool credentials and per-user grants.
- Which features need a separate license or product.
The MCP gateway buyer’s guide covers hosting, tenancy and evidence checks for the MCP side.
For the next design decision, MCP vs REST API shows how the tool layer can preserve an existing business API, while MCP vs function calling follows the host’s execution loop. Use the open source gateway comparison to shortlist a runtime. A managed deployment also needs the AI vendor risk assessment; SKILL.md instructions belong in a separate review from model and tool traffic policy.
Where does Jarvis Registry fit?
ASCENDING publishes this blog and maintains Jarvis Registry, an open source MCP registry and gateway. The original comparison sources were checked on October 25, 2026; the Jarvis permission and telemetry clarification below was checked against its technical documentation on October 28. The behavior still needs testing in your environment.
Jarvis Registry belongs on the MCP side of the table, rather than the application-facing model-routing side. Its security design separates application-role scopes from access to a particular server or agent. The auth guide documents Entra ID and Keycloak integration through its OIDC/OAuth layer. Check the actual identity-provider path rather than assuming native SAML support.
The tool-enablement contract documents a server-wide disabled-tool list, not a different tool policy for each user. Normal discovery and execute_tool enforce that list; the transparent proxy does not. The observability guide covers operational metrics and instrumented agent traces. Verify required identity, decision and tool-call records against real deployment output instead of inferring complete audit coverage from a product-page diagram.
Jarvis Chat is a separate employee chat workspace. Its page lists OpenAI, Anthropic and AWS Bedrock behind one interface, with access rules that scope models to users or teams. That’s a governed workspace, not an application-facing LLM gateway. If your applications need one, pair Jarvis with an LLM gateway and test that identity and trace IDs carry across both.
Bring one agent workflow that calls a model and a tool to a Jarvis demo, and run the five checks above against it. The Jarvis MCP gateway page lists the protocol standards it supports.
References
- a16z: How 100 Enterprise CIOs Are Building and Buying Gen AI in 2025
- Menlo Ventures: 2025 Mid-Year LLM Market Update
- Microsoft: AI gateway capabilities in Azure API Management
- AWS: Amazon Bedrock AgentCore Gateway
- LiteLLM: AI Gateway
- LiteLLM: MCP overview
- Portkey: AI Gateway
- Portkey: MCP Gateway
- Kong: AI Gateway
- Kong: AI MCP Proxy plugin
- TrueFoundry: AI Gateway
- Cloudflare: AI Gateway
- Cloudflare: MCP server portals
- MCP specification 2026-07-28: Sampling (deprecated)
LLM gateway and MCP gateway questions
What is the difference between an MCP gateway and an LLM gateway?
An LLM gateway sits between applications and model providers to govern model requests. An MCP gateway sits between AI clients and MCP servers to govern tool access. Routing, budgets, credential handling, per-tool authorization and logging depend on the product, edition and configuration; compare the exact controls your workflow requires.
Do I need both an LLM gateway and an MCP gateway?
Not automatically. Evaluate model-side and tool-side controls separately, then check what your existing clients, services or API platform already enforce. One product may cover both responsibilities, and a small direct integration may not need an additional gateway. When you use both layers, test the identity and evidence needed to investigate the complete request.
Can one product be both an LLM gateway and an MCP gateway?
Yes. On October 25, 2026, documentation from Azure API Management, Amazon Bedrock AgentCore Gateway, LiteLLM, Portkey, Kong and TrueFoundry described both model and MCP traffic. One product doesn't guarantee one policy model, so test whether identity, policy rules, logs and licensing cover both kinds of traffic. See the Azure API Management AI gateway overview for one example.
Is an AI gateway the same as an LLM gateway?
Not always. Vendors use "AI gateway" both for model-only gateways and for products that also govern MCP tools and agent-to-agent traffic. Azure API Management's AI gateway, for example, covers language model APIs, MCP servers and A2A agent APIs. Ask which traffic types a product governs rather than relying on the label.

