LLM Gateway Compared: LiteLLM vs Portkey vs OpenRouter vs Cloudflare vs Vercel

TL;DR: An LLM gateway is a proxy between your application and model providers that unifies the API, routes and retries requests, and tracks spend. LiteLLM and Portkey are open-source gateways you can self-host. OpenRouter, Cloudflare AI Gateway and Vercel AI Gateway are managed services. Self-host when data residency and key custody matter most. Use a managed gateway when you want failover without running a proxy.

Key Takeaways

• An LLM gateway solves four problems at one layer: one API format across providers, automatic fallback when a provider fails, central key custody, and per-team spend control.
• LiteLLM is the self-hosted default for Python-heavy platform teams: an OpenAI-format proxy for 100+ providers with virtual keys, budgets, and YAML-defined fallback chains.
• Portkey's MIT-licensed gateway is the lightest self-hosted option, a Node.js gateway with JSON routing configs, conditional fallbacks on status codes, and built-in guardrails.
• OpenRouter is a model marketplace: one key, hundreds of models, no markup on inference, a fee on credit purchases, and provider routing by price, latency or throughput.
• Cloudflare AI Gateway and Vercel AI Gateway are edge-hosted control planes. Cloudflare's core features are free with a 5% fee on unified-billing credits. Vercel adds zero token markup, including with BYOK.
• An LLM gateway is not an MCP tool gateway. AgentCore Gateway routes agent tool calls; LiteLLM and its peers route model calls. Production agent stacks usually need both.

Why do teams put a gateway in front of their LLM calls?

The first LLM feature in most codebases calls one provider SDK directly. That works until the second provider shows up. Maybe a cheaper model wins on a subtask, a region needs a different vendor, or an outage takes down the only model you depend on. At that point every call site has to learn a second API shape, a second retry policy, and a second way of counting tokens.

The timing matters in late 2026. Frontier labs now distribute their models through several clouds at once: OpenAI's open-weight models on AWS Bedrock and, per this week's announcement, OpenAI models on Google Cloud, plus Anthropic models on Bedrock and Vertex AI. That makes "which model" and "which provider serves it" two separate decisions. The same model can be reached through three or four endpoints with different prices, rate limits and data-handling terms. A gateway is the layer that makes that choice at request time instead of at code-review time.

Search data from our own readers points the same way. Queries like "pros and cons of model-agnostic llm gateway for small apps" and "agentcore gateway llm gateway" show people trying to work out which kind of gateway they actually need. This guide answers that with a side-by-side comparison of the five gateways most teams shortlist, plus a framework for when you need none of them.

What exactly does an LLM gateway do?

An LLM gateway is a reverse proxy that speaks an LLM API (almost always the OpenAI Chat Completions format, increasingly also the Anthropic Messages and OpenAI Responses formats) on the front and translates to each provider's native API on the back. Everything else is policy that runs on that request path:

            ┌──────────────────────────── LLM Gateway ────────────────────────────┐
 App  ───▶  │ auth (virtual key) → budget check → cache lookup → route selection │ ───▶ Provider A
 (OpenAI    │        ↓                                     ↓                     │ ───▶ Provider B
  format)   │   guardrails (pre)                retry / fallback / cooldown      │ ───▶ Self-hosted vLLM
            │        ↓                                     ↓                     │
            │   guardrails (post) ← response normalization ← usage + cost log    │
            └─────────────────────────────────────────────────────────────────────┘

The capabilities break down into five groups:

1. API unification. One request shape and one response shape, so changing models means changing a string, not a client library.
2. Reliability. Retries with backoff, fallback chains across providers and models, and cooldowns that stop routing to a deployment after repeated failures.
3. Key custody. Provider keys live in the gateway. Applications and developers get scoped virtual keys that can be rotated or revoked without touching provider accounts.
4. Cost control. Token and dollar accounting per key, team or project, with budgets that reject requests once a limit is hit.
5. Observability and safety. Request logs, latency and token metrics, optional caching, and pre- or post-call guardrails such as PII redaction.

None of these are hard to build for one provider. What makes a gateway worth having is getting all five right across every provider at once, including streaming, tool calls and the error-code quirks that differ between vendors.

How do LiteLLM, Portkey, OpenRouter, Cloudflare and Vercel compare at a glance?

Two rows decide most evaluations: deployment (who operates the proxy) and data path (whose network your prompts cross). Everything else is a feature delta you can usually work around.

How does LiteLLM work, and when is it the right choice?

LiteLLM ships in two forms. The Python SDK gives you litellm.completion() with one call signature across providers. The proxy server is the gateway: a standalone OpenAI-compatible service, configured through a config.yaml, that any language can call.

The proxy's strength is how much routing behavior you can express in configuration. Model groups are named aliases. Repeating a model_name across entries load-balances across deployments, and fallback chains map one group to others:

model_list:
  - model_name: chat-default            # alias your app calls
    litellm_params:
      model: bedrock/anthropic.claude-sonnet-4-5
      aws_region_name: us-east-1
  - model_name: chat-default            # same alias = load-balanced deployment
    litellm_params:
      model: vertex_ai/claude-sonnet-4-5
      vertex_location: us-east5
  - model_name: chat-backup
    litellm_params:
      model: openai/gpt-5-mini
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  enable_pre_call_checks: true          # required for context-window fallbacks
  fallbacks: [{"chat-default": ["chat-backup"]}]
  context_window_fallbacks: [{"chat-default": ["chat-backup"]}]

litellm_settings:
  num_retries: 3                        # per group, before falling back
  request_timeout: 60
  allowed_fails: 3                      # failures/minute before cooldown
  cooldown_time: 30

This example shows the pattern described above: the same model served from two clouds as one load-balanced group, with a different model as the fallback. Your application only ever sends model: "chat-default".

On top of routing, the proxy adds virtual keys, per-key and per-team spend tracking and budgets (backed by Postgres), callbacks to observability tools such as Langfuse, and an admin UI. LiteLLM publishes benchmark figures for proxy overhead (it cites 8 ms P95 at 1k RPS). Treat that as a vendor number to reproduce on your own hardware, not a guarantee.

Choose LiteLLM when you have a platform team, a Python-friendly ops stack, and requirements like "keys never leave our VPC" or "every team gets a budget". Watch out for the operational surface. You own Postgres, Redis (if you enable shared caching or rate limits), upgrades, and supply-chain hygiene. Pin exact versions and verify the signed Docker images LiteLLM publishes rather than pulling latest. A gateway holds every provider credential you own, so it is the highest-value target in your AI stack.

How does Portkey's gateway differ from LiteLLM?

Portkey's open-source AI Gateway (MIT licensed) runs on Node.js: npx @portkey-ai/gateway starts it locally, and it deploys to Docker, a plain Node server, or Cloudflare Workers. Routing is defined in JSON configs that can be attached to a request header or saved on the hosted platform:

{
  "strategy": { "mode": "fallback", "on_status_codes": [429, 503] },
  "targets": [
    { "override_params": { "model": "@openai-prod/gpt-5-mini" } },
    { "override_params": { "model": "@anthropic-prod/claude-sonnet-4-5" } }
  ]
}

By default, a fallback fires on any non-2xx response. on_status_codes narrows it, which matters more than it looks. You usually want to fall back on a 429 (rate limit) or 503 (overload), but a 400 (your request is malformed) will fail identically on every target. Retrying it just multiplies latency and cost. Targets can themselves be load-balancers or nested strategies, so you can build "load-balance across two keys, fall back to a second provider" as one tree.

Portkey's other differentiator is guardrails. The gateway has a large catalogue of pre-built checks (regex, JSON-schema, PII and others) that can run before or after a model call and can deny, flag or trigger a fallback. Semantic caching, the richer observability UI and team management live in the hosted and enterprise tiers.

Choose Portkey when you want a small self-hosted footprint, a TypeScript-native stack, or guardrails at the gateway rather than in each service. Choose LiteLLM instead when you need the deepest provider coverage and a self-hosted admin and budget system without a paid tier.

When does OpenRouter make more sense than a self-hosted gateway?

OpenRouter flips the model. You don't run a gateway; you buy credits and get one API key for a very large catalogue of models from many inference providers. It has two routing layers:

• Model fallback. A models array in the request lists alternatives in priority order.
• Provider routing. The provider object controls which upstream serves a given model: order, only, ignore, sort by price/latency/throughput, a max_price ceiling, and data-policy filters such as data_collection: "deny" and zdr: true for zero-data-retention endpoints.

{
  "models": ["anthropic/claude-sonnet-4.5", "openai/gpt-5-mini"],
  "messages": [{ "role": "user", "content": "Summarize this incident report." }],
  "provider": {
    "sort": "latency",
    "data_collection": "deny",
    "zdr": true,
    "max_price": { "prompt": 3, "completion": 15 }
  }
}

The economics are simple. OpenRouter states there is no markup on inference pricing. Its revenue is a fee on credit purchases, plus a 5% fee on bring-your-own-key usage above a monthly allowance. For open-weight models served by many competing providers, sort: "price" can land well below what one provider would charge.

Choose OpenRouter when you need breadth fast: prototyping across many models, evaluating open-weight models without signing up with each host, or a small app that wants failover with zero infrastructure. Avoid it as the only path when your contracts require prompts to stay inside your cloud account, or when you need enterprise discounts negotiated directly with a provider.

What do Cloudflare AI Gateway and Vercel AI Gateway add?

Both are managed control planes run by edge platforms. You keep calling a model API, but through their endpoint, and you get logging, caching and failover without deploying anything.

Cloudflare AI Gateway sits in front of Workers AI, OpenAI, Anthropic, Google Gemini and other providers. Its core features (analytics dashboard, caching and rate limiting) are free on all plans, and it supports request retries and model fallbacks. Logs are subject to per-plan storage limits. With unified billing, Cloudflare buys inference on your behalf at the provider's per-token rate and charges a 5% fee on purchased credits. Guardrails are billed as Workers AI inference. It fits best if your app already runs on Workers, because caching and rate limiting then sit on the same network as your code.

Vercel AI Gateway is callable from any infrastructure, not just Vercel deployments, through the AI SDK, OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages formats. It routes across providers with configurable provider order and model fallbacks, records every routing attempt in request logs, and supports budgets per team, project, API key or member. Vercel states it adds zero markup to provider token prices, including BYOK traffic. One detail to read before relying on it: budgets cover spend billed through Vercel's system credentials, while BYOK spend is metered separately. If a BYOK request fails, the gateway can fall back to system credentials, so check that behavior if "only use our provider contract" is a hard requirement.

Choose an edge gateway when your app already lives on that platform and you want observability and failover in an afternoon. Look elsewhere when you need custom routing logic, on-prem models behind your firewall, or a single audit log spanning several clouds you control.

Is an LLM gateway the same as AgentCore Gateway or an MCP gateway?

No, and mixing them up is the most common mistake we see in architecture reviews. Both are called "gateways", but they sit on different arrows of an agent diagram:

AgentCore Gateway turns Lambda functions and APIs into MCP tools that agents can discover and call with IAM-scoped credentials. It does not choose which model answers a prompt. A production agent typically needs both: an LLM gateway for the reasoning loop, and a tool gateway for actions, with deterministic authorization on the tool side. Some vendors now market a combined "AI gateway" for both. Evaluate the two halves separately anyway, because their failure modes don't overlap.

What are the pros and cons of a model-agnostic gateway for a small app?

This is the question behind the most specific search query we see on the topic, and the honest answer is that a small app often doesn't need a self-hosted one.

Pros, even at small scale:

• Outage insurance. One fallback entry turns a provider incident from downtime into slightly different answers.
• Model agility. When a cheaper model matches your quality bar, switching is a config change, not a refactor.
• One bill and one dashboard. Managed gateways consolidate spend that would otherwise be scattered across provider consoles.

Cons:

• An extra hop and an extra dependency. Your availability is now the gateway's availability multiplied by the provider's. A self-hosted proxy that you forgot to scale is a worse single point of failure than any provider.
• Lowest-common-denominator APIs. Provider-specific features (prompt caching controls, extended-thinking parameters, server-side tools, citations) arrive in gateways late or are flattened by the translation layer. If your product depends on one of them, test it through the gateway before committing.
• Another party in the data path. With a managed gateway, prompts cross a third network and fall under a third set of retention terms.

The pragmatic path for a small app: start with a provider-agnostic SDK rather than a gateway. The Vercel AI SDK or LiteLLM's Python SDK gives you one call signature with no proxy to run; our TypeScript LLM SDK guide compares the options. Add a managed gateway when you need cross-provider failover or spend visibility, and self-host only when key custody or data residency force it.

How should you choose an LLM gateway?

Work through four questions in order. Each one eliminates options:

1. Must prompts and keys stay inside your network? If yes, self-host: LiteLLM or Portkey OSS. Managed gateways are out, whatever their features.
2. Who will operate it? With no platform team, a managed gateway (OpenRouter, Cloudflare, Vercel) almost always beats a self-hosted proxy nobody upgrades. With one, LiteLLM's budgets and admin UI pay for their operational cost quickly.
3. Where does your app already run? On Workers, start with Cloudflare. On Vercel or the AI SDK, start with Vercel AI Gateway. In a Python service mesh, start with LiteLLM.
4. How many models do you really need? Two or three providers on enterprise contracts favor a self-hosted router over your own keys. Dozens of models for evaluation or open-weight cost-shopping favor OpenRouter.

Whatever you pick, test it the same way before production:

• Kill a provider. Point one deployment at an invalid key or a blackholed endpoint, then confirm traffic shifts and the switch shows up in logs.
• Exceed a context window. Send an oversized prompt and confirm it falls back to a larger-context model (or fails fast) rather than retrying the same error.
• Exhaust a budget. Set a $1 limit on a test key and confirm the next request is rejected, not silently billed.
• Stream a tool call. Run a streaming response with function calling end-to-end, because streaming plus tools is where translation layers break most often.
• Measure overhead. Record P50/P95 latency with and without the gateway on your real payload sizes.

Feed the gateway's logs into the same tracing backend as the rest of your agent. Our LangSmith vs Langfuse vs Phoenix comparison covers how to pick one. A fallback that fires silently is a cost and quality change you want to see.

Frequently Asked Questions

What is an LLM gateway?

An LLM gateway is a proxy between applications and model providers that exposes one API, usually OpenAI-compatible, and applies routing, retries, fallbacks, key management, caching and spend tracking to every request. It lets you change models or providers through configuration instead of code.

Is LiteLLM or Portkey better for self-hosting?

LiteLLM is better when you want the widest provider coverage and built-in virtual keys, team budgets and an admin UI in the self-hosted product. Portkey's MIT gateway is better when you want a lighter Node.js deployment, JSON routing trees with status-code-specific fallbacks, and gateway-level guardrails.

Does OpenRouter charge more than calling providers directly?

OpenRouter states it adds no markup to inference pricing. It charges a fee when you buy credits, and a 5% fee on bring-your-own-key usage above a monthly allowance. Because it can route to the cheapest provider for open-weight models, total cost can come out lower than a single direct provider.

Do I need an LLM gateway if I already use AWS Bedrock?

Not always. Bedrock already gives you many models behind one AWS API, plus cross-region inference. A gateway becomes useful when you also call providers outside Bedrock, need failover to another cloud, or want per-team virtual keys and budgets that span providers. AgentCore Gateway handles MCP tool calls, not model routing.

Does an LLM gateway add noticeable latency?

A well-run gateway adds single-digit to low tens of milliseconds, which is small next to model generation time measured in seconds. The bigger latency risks are misconfigured retries that repeat a non-retryable error, and an undersized self-hosted proxy. Measure P95 with your own payloads before and after adding it.

Can an LLM gateway route to self-hosted models like vLLM?

Yes. LiteLLM, Portkey and Cloudflare AI Gateway can all target any OpenAI-compatible endpoint, which includes vLLM servers. A common pattern is to route routine traffic to a self-hosted open-weight model and fall back to a hosted frontier model on errors or context-length limits.

Sources

• LiteLLM GitHub Repository - OpenAI-format proxy and SDK for 100+ providers
• LiteLLM Proxy Reliability - Fallbacks, retries and cooldowns
• Portkey AI Gateway GitHub Repository - MIT-licensed open-source gateway
• Portkey Fallbacks Documentation - Strategy configs and status-code triggers
• OpenRouter Provider Routing - Order, sort, data policy and price ceilings
• OpenRouter FAQ - Pricing, credit fees and BYOK
• Cloudflare AI Gateway Documentation - Caching, rate limiting and fallbacks
• Cloudflare AI Gateway Pricing - Free features and unified billing fee
• Vercel AI Gateway Documentation - Routing, budgets, BYOK and zero markup

Originally published at fp8.co. Subscribe for weekly AI engineering analysis at fp8.co/newsletters.