I Compared 5 LLM Gateway Tools for Real-World Production Use

When an LLM application starts handling real users, calling a model API is usually easy. The problems show up around it. One provider hits a rate limit, another slows down, an endpoint goes down, retries drive up costs, or the application needs to switch models without rewriting half the codebase.

This is where an LLM gateway becomes useful. It sits between your application and model providers, giving you a central place to manage routing, reliability, observability, and access. I compared five widely used options, LiteLLM, Kong AI Gateway, Portkey, Bifrost, and Apache APISIX, to see how their approaches differ when production concerns such as traffic management, latency, provider failures, and operational control start to matter.

What Is an LLM Gateway?

An LLM gateway is a software layer that sits between an application and one or more large language model providers.

Think of it as a traffic controller for AI requests.

Without a gateway, an application may need separate integrations for providers such as OpenAI, Anthropic, Google, Azure, Amazon Bedrock, or self-hosted models. Each provider can have different APIs, authentication methods, rate limits, error responses, model names, and usage policies.

That complexity can quickly spread across an application.

An LLM gateway provides a central layer where much of that provider-specific logic can live.

A typical request looks like this:

Application → LLM Gateway → Model Provider → LLM Gateway → Application

Depending on the platform, the gateway can handle:

• Model and provider routing
• Load balancing
• Rate limiting
• Retries
• Provider fallbacks
• Authentication
• Logging
• Token tracking
• Cost monitoring
• Request monitoring
• Access control
• Caching
• Usage policies

The exact feature set varies between gateways.

The important distinction is that an LLM gateway is not an AI model. It does not replace GPT, Claude, Gemini, or another model. Instead, it manages the infrastructure surrounding those models.

For a simple application using one provider, a gateway may not be necessary. For an application using several providers or handling substantial traffic, the centralized control can become much more valuable.

Why Do You Need an LLM Gateway in Production?

A direct model API integration can work perfectly well during development.

Things change when traffic becomes unpredictable.

Imagine an application receiving thousands of requests during a busy period. The primary model provider begins returning 429 rate-limit responses. Without a gateway, the application must decide whether to retry, wait, switch providers, or return an error.

With an LLM gateway, you can handle those policies in a centralized layer.
The same idea applies to model changes.

If provider-specific logic is scattered across several applications and services, switching providers can require significant development work. A gateway can provide a consistent interface while managing routing rules and provider configurations separately.

Visibility is another consideration.

When requests pass through a centralized gateway, engineering teams can gain a clearer view of latency, failures, token usage, provider behavior, and traffic patterns.

That information becomes particularly useful once an LLM application moves beyond experimentation.

A gateway can also help separate application logic from infrastructure decisions. Developers can focus on the application while platform teams manage routing, limits, authentication, and provider policies.

That does not mean every production application needs a gateway. If the application is small, uses one model provider, and has limited traffic, adding another infrastructure layer may create unnecessary complexity.

The value increases when provider diversity, traffic volume, reliability requirements, or governance become difficult to manage directly in the application.

How I Compared the LLM Gateways

I would not compare these tools purely on advertised requests per second.

A gateway can perform extremely well in a controlled benchmark and behave differently under a real application workload.

Production traffic introduces variables that simple benchmarks often miss. Requests can have different prompt sizes, responses can be streamed, traffic can arrive in bursts, and upstream providers can behave differently under load.

For this comparison, I focused on five areas.

Provider abstraction

How easily can an application work with different model providers?

A useful gateway should reduce the amount of provider-specific code developers need to maintain.

Traffic management

Can requests be routed, load-balanced, rate-limited, or directed according to defined rules?

This matters when applications use multiple deployments or providers.

Reliability

What happens when a provider times out, returns an error, or reaches a rate limit?

Retries and fallbacks can improve resilience, but they can also increase latency and cost.

Observability

Can engineers understand what is happening with requests?

Useful visibility includes latency, errors, tokens, costs, provider behavior, and request traces.

Operational control

How well does the gateway fit into the existing infrastructure?

A technically capable gateway may still be a poor fit if it introduces an operational model the engineering team does not want to maintain.

These criteria reveal differences that a single latency test cannot.

LLM Gateway Comparison at a Glance

Comparison of Leading Open-Source LLM Gateways

The five gateways take different approaches to the same underlying problem.

LiteLLM focuses heavily on multi-provider abstraction and centralized LLM management. Kong AI Gateway extends traditional API gateway capabilities into AI workloads. Portkey emphasizes routing, reliability, and observability. Bifrost takes a performance-focused approach to gateway infrastructure. Apache APISIX brings AI capabilities into a broader cloud-native API gateway ecosystem.

That makes the comparison more useful as an architectural evaluation than as a simple feature checklist.

LiteLLM

LiteLLM is built around a straightforward idea: applications should be able to interact with different LLM providers through a consistent interface.

Its Proxy Server provides a centralized gateway, while its Python SDK can also be used directly within applications. LiteLLM supports a large range of LLM providers and offers an OpenAI-compatible interface.

The platform also provides retries, fallbacks, spend tracking, authentication, rate limiting, logging, and monitoring.

Core strengths
The biggest strength is provider abstraction.

Instead of implementing separate application logic for every model provider, developers can use a common interface and move much of the provider-specific configuration into the gateway layer.

This becomes useful when a team is experimenting with several models or wants to avoid locking application code too tightly to one provider.

LiteLLM also provides routing capabilities for retries and fallbacks. When a deployment becomes unavailable or starts returning errors, routing rules can help determine what happens next.

Centralized controls are another useful part of the architecture.

Teams running multiple applications can manage authentication, budgets, rate limits, logging, and model access from a central layer rather than recreating those controls inside every service.

Best for
Best for teams that need broad model-provider support, a common API interface, and centralized management of LLM traffic.

The main production consideration is performance under the actual workload. If latency is critical, benchmark the full application path rather than relying on a generic gateway benchmark.

Kong AI Gateway

Kong approaches LLM traffic from the perspective of a broader API gateway.

That makes it relevant to organizations that already manage APIs through concepts such as authentication, routing, security, traffic control, and observability.

Kong AI Gateway supports multiple model providers and integrates AI traffic management into its wider gateway architecture.

Core strengths
Kong’s main strength is infrastructure-level traffic management.

Its AI Gateway supports routing, load balancing, streaming, authentication, access control, rate limiting, provider failover, and observability.

That combination becomes useful when LLM APIs are no longer an isolated application dependency and instead become part of a wider platform.

For example, an organization may already have API gateway policies for authentication, security, and traffic management. Bringing LLM requests into the same operational model can reduce the number of separate infrastructure systems engineers need to manage.

Kong also provides AI-specific visibility into requests, tokens, costs, and latency.

This lets you view AI traffic through the same infrastructure lens as other APIs.

Best for
Best for organizations that already use API gateway infrastructure and want AI traffic to follow established security, routing, governance, and observability patterns.

The trade-off is operational complexity. A broader infrastructure platform can require more gateway knowledge than a lightweight model proxy.

Portkey

Portkey focuses on the operational problems that become more visible when AI applications use multiple providers.

Its gateway provides a unified API and supports routing strategies such as load balancing, conditional routing, retries, and fallbacks.

Core strengths
Portkey’s approach centers on reliability and request-level visibility.

Suppose the primary provider returns an error. A configured fallback can route the request to another provider.

This can help applications recover from certain upstream failures without forcing every application service to implement its own provider-switching logic.

Portkey also provides visibility into requests and fallback attempts, which can make production troubleshooting easier.

Its load-balancing capabilities can distribute traffic between configured providers or deployments.

One important point to remember with fallbacks: they are not automatically free.

If the first request fails and another provider handles the request, the application can experience additional latency and model usage.

A fallback policy must balance availability, cost, and response time.

Portkey also provides logging, tracing, metadata, and metrics for understanding individual model requests.

Best for
Best for teams that prioritize routing, provider reliability, fallbacks, and detailed visibility into LLM requests.

Teams looking for a self-hosted, open-source gateway should also evaluate Portkey’s deployment and licensing model against their requirements.

Bifrost

Bifrost takes a performance-focused approach to LLM gateway infrastructure.

The project is written in Go and provides an OpenAI-compatible interface across multiple providers. Its documented integrations include OpenAI, Anthropic, AWS Bedrock, Google Vertex, Azure, Mistral, and Ollama.

Core strengths
Low gateway overhead and high-throughput infrastructure are central to Bifrost’s positioning.

Bifrost publishes performance benchmarks showing very low gateway overhead under specific test conditions. Those results can help you understand its performance goals, but you should not treat them as universal production guarantees.

Real-world latency depends on many factors, including hardware, concurrency, payload size, network conditions, streaming behavior, and the upstream model provider.

Bifrost also provides automatic failover, load balancing, semantic caching, request logging, analytics, and governance.

For teams processing large request volumes, minimizing infrastructure overhead can be an important part of the evaluation.

Best for
Best for teams that care strongly about gateway performance, high request volumes, and Go-based infrastructure.

Before deployment, test the exact providers and request patterns the application uses. This is especially important when the workload relies heavily on streaming or provider-specific API features.

Apache APISIX

Apache APISIX comes from the traditional cloud-native API gateway ecosystem.

That gives it a different position in the LLM gateway space. Instead of building an entirely separate infrastructure layer for AI, teams can extend an existing API gateway architecture to handle model traffic.

APISIX’s AI Gateway capabilities include model routing, load balancing, retries, fallback, token rate limiting, security, logging, and observability.

Core strengths
The biggest strength is flexibility within a broader API gateway ecosystem.

APISIX supports weighted model routing, health checks, token-based rate limiting, retries, fallback providers, and AI-specific observability.
This can help organizations already running APISIX for conventional APIs.

Instead of maintaining one gateway for normal application traffic and another specifically for AI requests, teams can potentially manage both through a common infrastructure layer.

The plugin architecture also gives infrastructure teams considerable control over how they handle traffic.

Best for
Best for cloud-native teams that want to manage traditional API traffic and LLM traffic through a common open-source gateway architecture.

The trade-off is complexity. The flexibility that makes APISIX useful can also mean more configuration and operational responsibility.

What Real Production Traffic Reveals

The biggest lesson from comparing LLM gateways is that production traffic changes the evaluation.

A development environment might send one request every few seconds.

Production can generate bursts of concurrent requests, long streaming responses, retries, provider rate limits, and sudden traffic-volume changes.

That is where gateway behavior becomes much more important.

For example, a provider might return a 429 during a traffic spike.

A retry could recover the request, but it can also increase latency and consume additional resources.

A fallback can improve availability, but sending the request to another provider may increase model usage and cost.

Load balancing introduces another consideration.

Distributing traffic between providers can reduce dependence on a single deployment, but providers can have different response times, prices, context limits, and model behavior.

That means routing is not simply an infrastructure decision. It can affect the application’s user experience and operating cost.

The same is true for caching.

Caching may reduce repeated model requests in workloads with predictable or repeated prompts, but you need to evaluate it carefully for applications where responses depend heavily on context or freshness.

Production testing should therefore reproduce the conditions that matter to the actual application rather than relying on one artificial traffic pattern.

What Should You Measure?

For a serious LLM gateway comparison, I would measure more than raw throughput.

p50 latency

This shows the typical request experience and provides a useful baseline for normal traffic.

p95 and p99 latency

Tail latency often matters more in production because it shows what slower requests look like during real workloads.

Error rate

Track errors under both normal and peak traffic. Also separate gateway errors from upstream provider errors where possible.

429 and timeout frequency

Rate limits and timeouts are common reasons for reliability problems in LLM applications.

Requests per second

Throughput helps establish how much traffic the gateway can handle under a particular configuration.

Gateway CPU and memory

A gateway should not consume disproportionate infrastructure resources simply to proxy model requests.

Token consumption

Token usage affects both performance and cost, particularly when retries and fallbacks are involved.

Provider fallback frequency

A high fallback rate can indicate provider reliability problems, routing issues, or an overly aggressive fallback policy.

Retry frequency

Measure retries because they can improve successful-request rates while also increasing latency and usage.

Cost per successful request

This is one of the most useful production metrics because a technically successful request is not necessarily an economically efficient one.

Streaming performance

Streaming applications should measure time to first token, sustained response behavior, interruptions, and completion time rather than treating the entire request as one latency number.

Most importantly, test failure scenarios.

A gateway that performs well while every provider is healthy tells you only part of the story.

A more revealing test is what happens when the primary provider slows down, returns errors, hits a rate limit, or becomes temporarily unavailable.

Which LLM Gateway Approach Fits Your Architecture?

The five gateways solve overlapping problems, but their priorities are different.

• LiteLLM is centered on provider abstraction and centralized LLM management.
• Kong AI Gateway connects LLM traffic with broader API gateway, security, and governance infrastructure.
• Portkey emphasizes routing, reliability, fallbacks, and request-level observability.
• Bifrost emphasizes performance and low gateway overhead.
• Apache APISIX provides an extensible cloud-native gateway approach that can cover both conventional APIs and AI traffic.

The useful question is therefore not simply, “Which gateway has the most features?”

A better question is:

Which gateway matches the application's infrastructure, provider mix, reliability requirements, and operational model?

A small application with one model provider may not need all of these capabilities.

A multi-provider AI platform with significant traffic may need considerably more control.

Conclusion

An LLM gateway becomes increasingly useful when an application has multiple model providers, meaningful production traffic, or reliability and observability requirements that are becoming difficult to manage inside application code.

The five gateways take different approaches to the same underlying problem.

LiteLLM focuses on multi-provider abstraction and centralized LLM management. Kong AI Gateway brings AI traffic into a broader API gateway model. Portkey emphasizes routing, reliability, fallbacks, and request visibility. Bifrost focuses strongly on gateway performance and throughput. Apache APISIX combines AI gateway capabilities with a broader cloud-native API infrastructure.

There is no useful shortcut around production testing.

Use representative traffic. Measure p95 and p99 latency. Trigger provider failures. Test rate limits. Monitor retries and fallbacks. Track token consumption. Measure gateway resource usage. Then calculate what each reliability mechanism actually costs.

That gives you a much clearer picture than a feature list or a single benchmark.

An LLM gateway should ultimately make model infrastructure easier to control, observe, and change without pushing that complexity into every application service.

Frequently Asked Questions (FAQs)

1. What is an LLM gateway?

An LLM gateway is a software layer between an application and one or more large language model providers. It can manage model routing, authentication, rate limiting, retries, fallbacks, logging, monitoring, token usage, and other operational tasks.

2. Why use an LLM gateway in production?

An LLM gateway can centralize provider integrations and traffic-management logic. It can make it easier to switch providers, handle certain upstream failures, control usage, monitor requests, and manage multiple models without duplicating infrastructure logic across applications.

3. What is the difference between an LLM gateway and an API gateway?

A traditional API gateway manages general application API traffic. An LLM gateway adds capabilities designed around model traffic, such as provider routing, token-based limits, model fallbacks, token tracking, and LLM-specific observability. Some platforms, including Kong and Apache APISIX, extend traditional API gateway architectures with AI-specific capabilities.

4. Which LLM gateways support multiple AI providers?

LiteLLM, Kong AI Gateway, Portkey, Bifrost, and Apache APISIX can all manage traffic involving multiple LLM providers, although provider coverage, integrations, deployment models, and capabilities differ. Teams should verify support for the exact models and APIs their application requires before deployment.

5. How do you test an LLM gateway for production?

Test the gateway with traffic that resembles the real application, including normal requests, peak concurrency, streaming responses, rate limits, timeouts, and provider failures. Measure p50, p95, and p99 latency; error rates; throughput; resource usage; retries; fallbacks; token consumption; and cost per successful request.