ASCENDING Blog Home
Jarvis AI
Cloud Services
Talent Solutions
Public Sector
About

MCP Gateway vs LLM Gateway: Which Layer Controls What

Read Time: 10 min read Publish Date:

Kelvin Yu portrait

Kelvin Yu

Contributing Writer · AWS Agents

Share on LinkedIn
On this page
Concept card comparing an LLM gateway, which governs model access, with an MCP gateway, which governs tool access, and the identity and trace that connect them

An LLM gateway controls the model-request path: which provider an application calls, which key it uses and how usage is limited. An MCP gateway controls the tool-request path between an AI client and an MCP server. The exact permissions, budgets and logs depend on the product and setup.

Evaluate both paths before buying another gateway. Your current client, application or API platform may already provide the controls you need.

The model side is already multi-provider. In a16z’s 2025 CIO survey, 37% of respondents used five or more models in production, up from 29% a year earlier (a16z, June 2025).

Switching providers was less common in a separate survey: 11% of teams changed provider in the past year, while 66% upgraded within their vendor (Menlo Ventures, July 2025). These are dated survey findings, not a measure of your team’s control gaps.

This guide is for architects and platform leads deciding which layer to buy first. It compares what each gateway controls, traces one agent request through both and lists products that claim both. For the MCP layer on its own, start with what an MCP gateway does and how to evaluate one.

Last verified: October 25, 2026, against the vendor documentation linked below and MCP spec revision 2026-07-28.

Key Takeaways

  • An LLM gateway governs model access; an MCP gateway governs tool access. Verify routing, permissions, credential handling and evidence in the selected product and connection mode.
  • The 2026-07-28 spec deprecates MCP sampling but does not remove it immediately. Plan the recommended provider-API migration and include server-originated model calls in your control review.
  • Several products now document both layers, including Azure API Management and Amazon Bedrock AgentCore Gateway. Test whether one identity, policy and trace really cover both.
  • Buy for a control gap you can demonstrate. Existing client, service or API-platform controls may already cover a small workflow.

Disclosure: ASCENDING builds Jarvis Registry, an open source MCP registry and gateway. Its public pages don’t describe an application-facing LLM gateway, and the comparison below is based on public documentation.

What does an LLM gateway control?

An LLM gateway controls how applications reach models. Applications send one request format to the gateway, and the gateway decides which provider and deployment serve it. That puts these controls in one place:

  • Routing and fallback. The gateway spreads load across deployments and fails over when a provider has a problem. LiteLLM documents load balancing, fallbacks and retries across 100+ providers.
  • Provider keys. Applications get a gateway key instead of each provider’s key, so provider secrets stay in one store.
  • Token budgets and rate limits. Azure API Management documents token-per-minute limits and token quotas per consumer. Portkey documents limits based on cost or tokens.
  • Caching. Semantic caches reuse answers to similar prompts to save tokens and time. Azure API Management and Portkey both document one.
  • Prompt and response guardrails. Filters can redact personal data or block content before it leaves your boundary. Kong AI Gateway documents prompt guards and a sanitizer that redacts personal data.
  • Usage logs. Cloudflare AI Gateway reports request, token and cost metrics.

If your main question is cost by team, see AI token cost tracking by users, teams and budgets. If it’s changing an application’s model, see switching AI models without rebuilding apps.

A gateway matters more as usage spreads. Anthropic held 32% of enterprise LLM API usage in mid-2025, OpenAI 25% and Google 20% (Menlo Ventures, July 2025). Each provider means another key, another bill and another log format without a gateway in front.

Bar chart of enterprise LLM API usage share in mid-2025: Anthropic 32%, OpenAI 25%, Google 20%, Meta Llama 9%, from Menlo Ventures

Source: Menlo Ventures survey of 150+ technical decision-makers, June 30 to July 10, 2025. Usage share is weighted by application scale.

What does an MCP gateway control?

An MCP gateway controls which tools agents can reach and what happens when they call one. A tool call requests an operation, such as reading a ticket or updating a record. Check who may run that operation and which account reaches the business system.

Authentication remains a concern. A May 2026 study found that 40.55% of its 7,973 live remote MCP servers exposed tools without authentication (Zhou et al., arXiv). That finding describes the study’s sample; it does not establish the state of every server today.

  • Discovery. The gateway shows each user only the tools they may use. Cloudflare’s MCP server portals list a server only for users who match an Allow policy.
  • Per-tool authorization. Kong’s AI MCP Proxy evaluates deny lists before allow lists at default and per-tool levels. It’s an AI Gateway Enterprise feature that needs Kong Gateway 3.12 or later.
  • Tool credentials. Amazon Bedrock AgentCore Gateway handles OAuth flows, token refresh and credential injection for each tool.
  • Protocol bridging. The same AWS service converts APIs and Lambda functions into MCP tools.
  • Call records. Kong writes every allowed and denied access attempt to an audit log. Portkey’s MCP gateway records each tool call with user identity, parameters and response.

Tool access also depends on how users sign in to the gateway. For the identity setup, see MCP OAuth with Entra ID or Okta.

MCP gateway vs LLM gateway, side by side

The two layers can share a vendor, but they inspect different traffic and hold different secrets. The rows below are evaluation areas, not a guarantee that every product implements every control.

RowLLM gatewayMCP gateway
Traffic it seesPrompts and completions sent to model providersTool discovery and tool calls sent to MCP servers
Typical callerApplications, agents and MCP servers that need a modelAI clients and agents acting for a user
Deployment modelSaaS, self-hosted proxy or part of a cloud API gatewaySaaS, self-hosted gateway or part of a cloud API gateway
Identity and access controlGateway keys or tokens per app, team or user; model allow listsUser sign-in; per-server and per-tool permissions
Credentials it holdsModel provider API keysTool and SaaS credentials, often per user
Audit loggingRequests, tokens, cost, latency, sometimes promptsWho called which tool, with which result
Supported MCP transportsNone; it speaks provider HTTP APIsStreamable HTTP; some also SSE or stdio (LiteLLM documents all three)
HostingNear applications and model endpointsNear tool servers and company systems
Cost controlsToken budgets, quotas and cachingRate limits on calls; tool costs live in downstream systems
Content guardrailsPrompt and response filtersArgument checks and tool allow lists
If it failsModel calls stop or fall backAgents lose tools; actions stop

How does one agent request pass through both gateways?

In an architecture that uses both layers, one agent request can cross the gateways more than once. Here is an illustrative support-agent workflow:

Diagram of one agent request crossing both gateways: the agent calls the model through the LLM gateway, calls a tool through the MCP gateway, then sends the tool result back to the model, with the same user identity and trace ID on every hop

Illustrative architecture. Correlated identity, authorization and trace records are requirements to demonstrate, not verified behavior of every gateway or deployment.

  1. The agent asks a model what to do. The request goes through the LLM gateway, which picks the provider, checks the budget and filters the prompt.
  2. The model asks for a tool. The agent sends that tool call to the MCP gateway, not to the model provider.
  3. The MCP gateway checks the user. It confirms the user may call this tool, injects the tool credential and records the call.
  4. The result goes back to the model. The next model call carries the tool output, so it passes through the LLM gateway again.
  5. The agent answers. One user request has produced at least two model calls and one tool call.

For an agent acting for an employee, retain enough evidence to connect that employee, the agent and the downstream action. A shared service key alone cannot identify the person behind the request. An autonomous workload instead needs its own accountable identity. In either case, test whether a request or trace ID links the model and tool records; do not assume that using one vendor correlates them automatically.

MCP sampling lets a server request a model completion through the client. The 2026-07-28 spec deprecates sampling but retains it for at least twelve months. It advises new implementations not to adopt it and existing implementations to migrate to provider APIs.

Deprecation does not change a running server’s route. Inventory any model calls the server makes itself, then decide where to control their keys, spend, data flow and logs.

Which products cover both layers?

Several products now document more than one kind of traffic. We reviewed each vendor’s own documentation on October 25, 2026 and recorded which traffic types it describes.

ProductModel trafficMCP trafficAgent-to-agent (A2A)
Azure API Management AI gatewayYesYes: expose REST APIs as MCP servers or pass through existing onesYes: import A2A agent APIs
Amazon Bedrock AgentCore GatewayYes: model-based routing across providersYesYes: passthrough targets
LiteLLMYesYes: Streamable HTTP, SSE and stdioYes
PortkeyYesYes: documented as its MCP GatewayNot described on the pages reviewed
Kong AI GatewayYesYes: AI MCP Proxy, Enterprise licenseMentioned
TrueFoundry AI GatewayYesYesNot described on the page reviewed
Cloudflare AI GatewayYesNot on the AI Gateway page; MCP server portals are a separate Cloudflare One featureNot described

Methodology. We read each product’s main gateway page and its MCP page where one exists. We recorded only what the vendor’s own pages state. We didn’t test the products, and a “Yes” says nothing about depth, licensing or maturity.

The trend is clear even in this small sample: the labels are converging. AWS now describes AgentCore Gateway as a single entry point for tools, agents and models. That doesn’t make the controls the same. One product can still keep separate keys, policies and logs for model traffic and tool traffic. When a product claims both, ask it to show one user, one policy change and one trace across a model call and a tool call.

Where does an agent gateway fit?

An agent gateway governs calls from one agent to another, usually over the A2A protocol. It’s a third kind of traffic: the caller delegates a whole task, not a single model call or tool call. Azure API Management and AgentCore Gateway both list A2A alongside models and tools. For how agent gateways differ from the API gateway you already run, see agent gateway vs API gateway.

Do you need an LLM gateway, an MCP gateway or both?

Start with the control gap in your actual traffic. Tool access and model access both need deliberate controls, but neither automatically requires another standalone gateway.

Find which path is failing

Consider a support assistant that answers a question and then looks up an invoice. The following are possible diagnostic cases, not reported test results:

What you observeWhere to startWhat to establish
The model provider rejects the request for exceeding a quotaModel-request pathWhich provider or gateway limit was reached, and which application owns the usage
The model responds, but the invoice tool denies accessTool-request pathWhich server grant and downstream identity were used; changing model routing will not grant invoice access
Both requests succeed, but the answer uses the wrong invoiceSource and task evaluationWhether the tool returned the right record and the model used it correctly; successful routing alone does not prove answer quality

Keep the request IDs and observed error with the finding. Choose a new control only after identifying what the existing path cannot do.

Choose the control, then the product

Your situationControl decision to evaluate
Applications call models; no agents use company tools yetCompare required budgets, routing and evidence with the model-provider and application controls already available
Agents use a few tools through one client, such as an IDECheck whether client and server controls suffice; evaluate an MCP gateway when you need shared discovery, identity or resource policy
Agents call models and company tools in productionEvaluate controls on both paths; one platform or separate gateways may fit, with demonstrated identity and evidence correlation
MCP servers call models themselvesInclude those model requests in the budget, credential and data-boundary design; route through an LLM gateway if that is the chosen control point
Agents delegate work to other agentsAssess A2A identity, authorization and task-routing requirements; verify what the existing platform supports

Before choosing one product for both layers, ask for five things in a demo:

  1. The same user identity on a model call and a tool call, not a shared service key.
  2. One policy change, such as removing a user from a group, that takes effect on both.
  3. One trace ID that links the model calls to the tool call.
  4. Where each credential lives: provider keys, tool credentials and per-user grants.
  5. Which features need a separate license or product.

The MCP gateway buyer’s guide covers hosting, tenancy and evidence checks for the MCP side.

For the next design decision, MCP vs REST API shows how the tool layer can preserve an existing business API, while MCP vs function calling follows the host’s execution loop. Use the open source gateway comparison to shortlist a runtime. A managed deployment also needs the AI vendor risk assessment; SKILL.md instructions belong in a separate review from model and tool traffic policy.

Where does Jarvis Registry fit?

ASCENDING publishes this blog and maintains Jarvis Registry, an open source MCP registry and gateway. The original comparison sources were checked on October 25, 2026; the Jarvis permission and telemetry clarification below was checked against its technical documentation on October 28. The behavior still needs testing in your environment.

Jarvis Registry belongs on the MCP side of the table, rather than the application-facing model-routing side. Its security design separates application-role scopes from access to a particular server or agent. The auth guide documents Entra ID and Keycloak integration through its OIDC/OAuth layer. Check the actual identity-provider path rather than assuming native SAML support.

The tool-enablement contract documents a server-wide disabled-tool list, not a different tool policy for each user. Normal discovery and execute_tool enforce that list; the transparent proxy does not. The observability guide covers operational metrics and instrumented agent traces. Verify required identity, decision and tool-call records against real deployment output instead of inferring complete audit coverage from a product-page diagram.

Jarvis Chat is a separate employee chat workspace. Its page lists OpenAI, Anthropic and AWS Bedrock behind one interface, with access rules that scope models to users or teams. That’s a governed workspace, not an application-facing LLM gateway. If your applications need one, pair Jarvis with an LLM gateway and test that identity and trace IDs carry across both.

Bring one agent workflow that calls a model and a tool to a Jarvis demo, and run the five checks above against it. The Jarvis MCP gateway page lists the protocol standards it supports.

References

LLM gateway and MCP gateway questions

What is the difference between an MCP gateway and an LLM gateway?

An LLM gateway sits between applications and model providers to govern model requests. An MCP gateway sits between AI clients and MCP servers to govern tool access. Routing, budgets, credential handling, per-tool authorization and logging depend on the product, edition and configuration; compare the exact controls your workflow requires.

Do I need both an LLM gateway and an MCP gateway?

Not automatically. Evaluate model-side and tool-side controls separately, then check what your existing clients, services or API platform already enforce. One product may cover both responsibilities, and a small direct integration may not need an additional gateway. When you use both layers, test the identity and evidence needed to investigate the complete request.

Can one product be both an LLM gateway and an MCP gateway?

Yes. On October 25, 2026, documentation from Azure API Management, Amazon Bedrock AgentCore Gateway, LiteLLM, Portkey, Kong and TrueFoundry described both model and MCP traffic. One product doesn't guarantee one policy model, so test whether identity, policy rules, logs and licensing cover both kinds of traffic. See the Azure API Management AI gateway overview for one example.

Is an AI gateway the same as an LLM gateway?

Not always. Vendors use "AI gateway" both for model-only gateways and for products that also govern MCP tools and agent-to-agent traffic. Azure API Management's AI gateway, for example, covers language model APIs, MCP servers and A2A agent APIs. Ask which traffic types a product governs rather than relying on the label.

Kelvin Yu portrait

About the author

Kelvin Yu

Contributing Writer · AWS Agents

Covers AWS-native agent infrastructure: Bedrock, AgentCore Runtime, and the deployment patterns that survive enterprise security review.

Keep reading