"AI Gateway vs API Gateway: They Solve Different Problems

TL;DR: An API gateway manages HTTP traffic between services - auth, routing, rate limiting, load balancing for REST and gRPC. An AI gateway manages LLM workloads — token-based rate limiting, model routing, cost attribution, semantic caching, guardrails. Use an API gateway for your microservices. Use an AI gateway for your LLM traffic. Most production teams eventually need both, sitting at different layers. This post walks through exactly where each one fits.

When we started adding LLM features to our platform, we already had Kong running for our microservices. The instinct was natural: route the LLM traffic through Kong too. Same auth, same rate limiting, same observability stack. One gateway to rule them all.

It worked for about six months, and only in the sense that requests got through. What it didn't give us was anything useful for actually managing AI workloads. We had no idea what each team was spending on tokens. We had no way to set a budget cap that would fire before the bill arrived. Our rate limits were based on requests per minute, which meant a single request with a 50k token prompt counted the same as one with a 200 token prompt. And when OpenAI had a partial outage, Kong had no concept of "try Anthropic instead" - we just served errors.

None of that is a criticism of Kong. It's doing exactly what it was designed to do. The problem was us expecting an API gateway to handle a fundamentally different category of infrastructure problem.

Here's the precise distinction, and why it matters architecturally.

What an API gateway actually does

An API gateway is a reverse proxy that sits between client applications and backend services. It handles the cross-cutting concerns of service-to-service HTTP communication: authentication, authorization, rate limiting, load balancing, SSL termination, request transformation, and routing based on URL paths or headers.

A typical request flow through an API gateway:

1. Client sends a request to the gateway endpoint
2. Gateway verifies authentication (API key, JWT, OAuth token)
3. Gateway applies rate limiting - is this client over their request quota?
4. Gateway routes to the appropriate upstream service based on the path
5. Gateway applies any request/response transformations
6. Response returns through the gateway back to the client

The unit of everything here is the HTTP request. Rate limits are requests per second or per minute. Cost is measured in compute, not in content. The gateway doesn't know or care what's inside the request body — it's routing bytes.

An API gateway works best when:

• You have multiple backend microservices that all need consistent auth and rate limiting
• You want a single ingress point for all your REST or gRPC traffic
• You need request/response transformation or protocol translation
• You're enforcing service mesh policies across a Kubernetes cluster
• Your traffic is synchronous and stateless

Kong, NGINX, AWS API Gateway, Apigee - these are all pretty good at this. They've been hardened in production for over a decade.

What an AI gateway actually does

An AI gateway is middleware that sits between your application code and your LLM providers. It handles the cross-cutting concerns of LLM traffic: model routing, token-based rate limiting, cost attribution, semantic caching, fallback chains, guardrails, and prompt/response logging.

A typical request flow through an AI gateway:

1. Application sends a prompt to the gateway's OpenAI-compatible endpoint
2. Gateway authenticates the request and checks the team/user's token budget
3. Gateway applies token-based rate limiting - is this team over their token quota?
4. Gateway routes to the appropriate model based on your rules (cost, latency, fallback)
5. Gateway applies input guardrails (PII detection, prompt injection checks)
6. Request goes to the LLM provider
7. Gateway applies output guardrails on the response
8. Gateway logs the full request with token count, cost, latency, user attribution
9. Response returns to the application

The unit here is the token, not the HTTP request. A single HTTP request might contain 50k tokens and cost $0.75. Another might contain 200 tokens and cost $0.003. Request-level rate limiting treats these identically. Token-level rate limiting doesn't.

An AI gateway works best when:

• You're routing traffic to one or more LLM providers and need unified observability
• Multiple teams or applications are hitting LLMs and you need cost attribution
• A model going down should trigger automatic failover, not user-facing errors
• You need to enforce spending limits before bills arrive, not after
• Guardrails on prompts or responses are a compliance or safety requirement
• You're deploying agents that call tools and need governed MCP access

The core difference, precisely stated

Both gateways manage traffic. The distinction is what dimension of that traffic they understand.

An API gateway understands HTTP - paths, methods, headers, authentication tokens.

An AI gateway understands LLM semantics - token counts, model capabilities, cost per inference, prompt content, response safety.

This is why you can't substitute one for the other. You could technically route LLM requests through Kong - it will auth them, rate limit them (by request count), and route them to OpenAI. But it cannot tell you that your data science team spent $847 on Claude Opus last week, or automatically switch to GPT-4o-mini when Claude is rate-limiting you, or catch a prompt injection attempt before it reaches the model. Those capabilities require a layer that understands what an LLM request actually is.

The mistake we made and the one you're probably making

We assumed token-level rate limiting was something we could implement in Kong with a plugin. We spent two weeks on a Lua plugin that tried to parse the OpenAI request body, extract token estimates, and apply limits. It technically worked in dev. It broke under streaming responses (tokens arrive incrementally - you don't know the total until the stream ends). It broke when we added Anthropic, whose request format differs. It broke when we added our self-hosted model, which had no token counting at all.

The deeper issue: we were building an AI gateway inside an API gateway. Every capability we needed - semantic caching, model fallback, token-based rate limiting, per-team cost attribution - would have required a custom plugin. The end state of that path was a custom AI gateway implemented as a Kong plugin collection, maintained by our team, with all the operational overhead that implies.

At some point the honest question is: why build this when it already exists?

The answer for us was switching to a dedicated AI gateway for LLM traffic, keeping Kong for everything else. The Kong configuration stayed exactly the same. The LLM services started routing through the AI gateway instead of directly to providers. We set OPENAI_BASE_URL and ANTHROPIC_BASE_URL in each service's config, pointed them at the AI gateway endpoint, and got token-level observability without touching application code.

Do you need both, or just one?

If you only have microservices and no LLM workloads: you need an API gateway. A dedicated AI gateway adds nothing.

If you have LLM workloads but no microservices (rare): you need an AI gateway. A generic API gateway won't handle what you need.

If you have both (most teams at this point): you need both, running at different layers. The API gateway handles your service mesh. The AI gateway handles your LLM traffic. They don't compete — they sit in different parts of the stack.

The practical setup: your AI gateway is one more upstream service from the perspective of your API gateway. External traffic → API gateway → your services → AI gateway → LLM providers. Each layer handles what it's designed for.

What we use and why

We kept Kong for our service mesh - it handles auth and routing for our ~30 internal services and we have years of operational investment in it.

For LLM traffic, we moved to TrueFoundry's AI Gateway. The specific things that mattered:

Token-based rate limiting in-memory. Auth, budget checks, and rate limits all run in-memory before the request goes out - no external DB lookup per request, no Redis dependency that can fail open. The benchmarks show 350+ RPS on 1 vCPU with under 10ms overhead, which we've matched in our own testing.

Per-team budget enforcement on the hot path. When a team hits their token budget, subsequent requests return a rate-limit error. Not an alert after the fact — an error before the spend happens. That's the architectural difference that fixed our 2am billing alerts.

Model fallback chains. We configured: Claude Opus 4 → Claude Sonnet 4.6 → GPT-4o → GPT-4o-mini, in priority order. When Claude's rate limits fire, traffic automatically routes to the fallback. Our apps stopped serving errors during model outages.

Unified cost attribution. One dashboard showing spend by team, by model, by application, including our self-hosted Llama deployment. Previously three separate billing dashboards with no cross-referencing.

The honest tradeoff: If you want something lighter, LiteLLM covers the core routing and cost tracking with lower infrastructure overhead — the SSO and team-level governance features require the enterprise license, but for smaller setups it's a solid starting point.

What's your current setup, are you routing AI traffic through a general API gateway, or have you added a dedicated layer? Particularly curious whether teams on Apigee or AWS API Gateway have found native AI features worth it, or whether they ended up with a separate AI gateway layer regardless. Drop it in the comments.