Multi-LLM Routers: Designing for Quality, Latency, and Fallback

Wiring multiple LLM providers into the same application looks attractive on paper: if one goes down another takes over, cheap models handle easy jobs, and the expensive model is saved for hard ones. In practice, every additional provider is an additional failure source, and a badly designed fallback chain can end up more fragile than a simple single-provider setup. In this post I walk through the core axes of a multi-LLM router — quality, latency, resilience — in the light of a production outage that happened to me, and tie each design decision to concrete parameters.

A Real Incident: Six Providers Dying on the Same Night

For a long time, this blog's content pipeline ran on a fallback ladder built over the free tiers of six separate LLM provider paths. On the night of August 23, 2026, the logs of a single generation run at 23:30 UTC read like this:

cerebras/gpt-oss-120b        → 402 "Payment required to access this resource"
cloudflare/llama-3.3-70b     → 401 {"code":10000,"message":"Authentication error"}
groq/openai/gpt-oss-20b      → 429 "Rate limit reached for model"
mistral/mistral-small-latest → reviewer JSON cut off mid-output
openrouter/nemotron-3-super  → off-schema output (no JSON object returned)
gemini (3 keys × 9 models)   → daily quota: 30 (key, model) pairs closed

All six provider paths were down at the same time — one demanding billing, one failing authentication, one hitting rate limits, two unable to produce output in the required shape, and the main engine long past its daily quota. The system published nothing that night. The lesson is simple but expensive: N free providers that share no guarantee whatsoever are not redundancy; they are N separate failure sources. Router design should start from that lesson.

Redundancy Is Measured, Not Counted: Failure-Domain Independence

I called the table above "six providers," but real redundancy accounting is not done by counting providers. Redundancy is measured by the number of shared failure domains, not by the number of providers. Two paths only back each other up if they diverge across most of these dimensions: provider, billing account, quota scope, region, model family, inference backend, network path, and authentication domain.

My own incident is the textbook example — I wrote about quota fail-over discipline in an earlier post, but I had not yet seen the real lesson back then. I had three separate Gemini API keys and treated them as three independent quotas. Google's Gemini API documentation is perfectly clear: "Rate limits are applied per project, not per API key" — and the daily request quota (RPD) resets at midnight Pacific time. If the keys live in the same project, three keys are three doors into a single quota; adding doors does not enlarge the warehouse. The same illusion appears with model aggregators: "model X via OpenRouter" and "model X direct from the provider" sometimes land on the same inference backend, even the same region — two paths on paper, one failure domain in practice.

That is why long-lived quota/quarantine records should not be keyed by (api_key, model) but by the scope where the quota is actually enforced: (provider, project, model, quota_dimension). As a general rule: do not mistake API key diversity for quota diversity; confirm from the documentation at which scope the provider enforces its limits (key, project, organization, account).

Routing Strategy: The Quality, Latency, Cost Triangle

The router's first job is answering "which deployment should this request go to right now?" One of the most widely used open-source tools in this space, LiteLLM (an AI gateway exposing 100+ providers behind a single interface, with 57k GitHub stars), answers with several strategies, and they generalize to any router design:

• simple-shuffle (default): picks deployments randomly, weighted by RPM/TPM or explicit weights. It carries the least latency overhead, and the documentation recommends it as the default.
• latency-based-routing: caches recent response times and picks the fastest deployment. The lowest_latency_buffer parameter keeps candidates within a percentage band of the fastest one in the pool, preventing a single deployment from being hammered.
• usage-based-routing / rate-limit-aware: tracks TPM/RPM consumption via Redis and filters out deployments that are close to their limits.
• least-busy: picks the deployment currently handling the fewest active calls.
• cost-based-routing: picks the cheapest deployment based on input_cost_per_token and output_cost_per_token.

The practical rule for choosing a strategy: you cannot optimize an axis you do not measure. Latency-based routing needs a window of recorded response times; usage-based routing needs a shared counter (Redis). If you are not going to run that infrastructure, simple-shuffle plus correct RPM/TPM declarations is enough for most workloads — and it is the easiest to operate.

One warning on the cost axis: token price is not request cost. If the cheap model needs three attempts, fails validation twice, and triggers a fallback, it is effectively more expensive than the pricey model that finishes in one attempt. The real unit of comparison is quality-adjusted expected cost: inference cost + retry cost + fallback cost + failure penalty. Once you start logging that total per deployment, your "cheapest model" ranking usually changes.

The quality axis comes before the routing strategy: not every model fits every job. The real fragility in our incident was using cheap models in the reviewer role (structured JSON evaluation) — mistral-small produced truncated JSON, nemotron produced off-schema output. The reviewer role demands a narrower but stricter contract than the writer role: output must match a schema, fields must be complete, verdicts must be reproducible. That is why model pools should be separated by request type (generation ↔ evaluation ↔ summarization ↔ translation) and why each role's minimum capability bar should be written down. A bar like "any model in the reviewer pool must support schema-guaranteed structured output" saves you from spending your morning cleaning up broken JSON produced by a small model that got promoted to reviewer duty simply because it was idle last night.

Latency Is Not One Number: TTFT and p95

"Latency" is in the title, but LLM latency does not fit into a single number. There are at least four separate metrics: queue latency, time to first token (TTFT), generation latency, and total latency. In a streaming interface, user experience is mostly determined by TTFT: if the first token arrives in 800 ms and the full answer takes 25 seconds, the experience feels fluid; if the first token takes 12 seconds and the total is 15, the "faster" second path feels frozen to the user.

The second trap is using averages. Provider latencies are typically heavy-tailed: 900 ms, 950 ms, 1.0 s, 1.1 s — and then an 18-second request slips in. The average hides that tail; percentiles do not. For router health, p50 is a familiar reference, but the decision metric should be p95 (p99 if you need it). "Provider X averages 2.1 seconds" says little; "p50 900 ms, p95 8.2 s, error rate 1.8%" can feed a routing decision directly.

There is one technique at the edge where paying for latency is genuinely worth it: the hedged request. You send the request to the primary deployment; if no response has started by your threshold (say 750 ms), you start the secondary in parallel, take whichever answers first, and cancel the other. The cost is obvious: some requests are billed twice. That is why hedging is reserved for user-facing low-latency paths; background batch generation gets by with normal fallback.

A Fallback Taxonomy: Not All Errors Are Equal

The second design decision: what happens when a request fails? The most common mistake here is throwing every error type into a single "retry it" bucket. LiteLLM's fallback model forces the distinction with three separate configurations:

litellm_settings:
  # For general errors (rate limit, timeout, 5xx)
  fallbacks: [{"gpt-3.5-turbo": ["gpt-4"]}]
  # A separate chain for content policy refusals
  content_policy_fallbacks: [{"claude-2": ["my-fallback-model"]}]
  # A wider-context model for context window overflows
  context_window_fallbacks: [{"gpt-3.5-turbo-small": ["gpt-3.5-turbo-large"]}]
  # Last resort if a model group is misconfigured
  default_fallbacks: ["claude-opus"]

The reason for the split is mechanical: a context window overflow never heals by retrying the same model — you must move to a model with a wider window. Content policy refusals behave the same way: the same request will be refused again by the same model.

A 429, meanwhile, splits in two, and skipping that split is expensive. A burst limit (per-minute RPM/TPM overrun) is genuinely transient: respect the Retry-After header or the provider's quota metadata and try again in a few seconds. Daily quota exhaustion arrives with the same HTTP code but is an entirely different animal: retrying before the reset moment (midnight Pacific for Gemini) is pointless. The way to tell them apart is not the HTTP code but the response body and the Retry-After value — same code, different semantics.

Which brings us to a more general principle: an HTTP status alone is not an error taxonomy. A 401 can be a wrong key — or an expired token, a freshly rotated secret that has not propagated yet, or a transient credential deployment issue. A 404 can mean the model was truly retired — or a wrong region or a wrong deployment name. Robust classification uses four inputs: HTTP status + the provider's own error code + the response body + that deployment's historical behavior.

The taxonomy, summarized:

Retry Discipline: Recovery or Amplification?

The most dangerous side effect of a fallback chain is retry amplification, covered in detail in the "Addressing Cascading Failures" chapter of Google's SRE book. The book's warning is numeric: retries multiply across layers — if each of three layers independently retries 3 times, the service at the bottom can see 4³ = 64 requests for a single user request. LLM routers are especially prone to this trap because the chain is usually multi-layered: application retries × router retries × provider SDK retries.

The three disciplines the book recommends carry over to the LLM context verbatim:

1. Randomized exponential backoff ("always use randomized exponential backoff when scheduling retries"): without jitter, all clients come back at the same moment after a network blip and recreate the problem themselves.
2. A retry budget: a fixed per-process retry ceiling per minute (the book's example: 60/minute). When the budget is exhausted, the request is not retried and the error propagates up.
3. Retriable vs. non-retriable separation: under overload the server should say so with an explicit code, and the client seeing that code should not retry.

At the router layer, the practical translation is: consolidate retries into a single layer (preferably the router), disable application and SDK retries, and budget total attempts per run, not per request. There is a nice real-world embodiment of this principle: Cloudflare AI Gateway lets you hand retry behavior to the gateway via request headers — cf-aig-max-attempts (up to 5 attempts), cf-aig-retry-delay, and cf-aig-backoff (constant / linear / exponential). The application does not retry, the SDK does not retry, the gateway does: retries get a single owner.

Two more concerns intertwine with retries. The first is timeouts: a request without a timeout means one slow provider can swallow the entire retry budget in a single attempt. Because LLM response time grows with output length, timeouts should be tuned per request type (on the Cloudflare side, that is the cf-aig-request-timeout header), and the sum of timeout plus retry intervals must not exceed the time budget of the layer above.

The second, and the most often skipped: idempotency. Consider this scenario: the provider produced a response, the connection dropped at that exact moment, the router treated it as a timeout and moved to the second deployment. Now there are two responses. If the rest of the chain has side effects — publishing, e-mail, a database insert, billing — "one retry" silently becomes "two jobs." The fix is a request_id / idempotency key carried end to end, and a line drawn clearly: LLM inference retries and business operation retries are not the same thing — the former can repeat freely; the latter may repeat only under the protection of an idempotency key.

A Circuit Breaker Is a State Machine, Not a List

The "it failed, blacklist it" reflex is a caricature of a circuit breaker. A real circuit breaker is a three-state machine:

• HEALTHY: traffic flows normally. If the error threshold within a window is exceeded (say 5 errors in 60 seconds) → OPEN.
• OPEN: the deployment is closed to traffic. After a period (say 5 minutes) or when a condition changes → HALF_OPEN.
• HALF_OPEN: a single probe request is sent. Success → HEALTHY; failure → back to OPEN.

The critical subtleties live in the transitions. Permanently killing a provider because you saw a 401 is too harsh: five minutes later the secret rotation completes and the credential is fine — but your router has buried it forever. The right move is tying the exit from OPEN to events: a timeout, a configuration version change, a credential update, or a scheduled probe. For quota exhaustion, the OPEN duration is not a guess but a known reset moment — if Gemini's daily quota opens at midnight Pacific, you stamp that timestamp on the OPEN record and the machine wakes itself up. Short-term cooldowns (in LiteLLM, allowed_fails, default 3, and cooldown_time, default 5 seconds) are the lightweight version of this machine's HEALTHY→OPEN edge; the persistent state file is the OPEN state on disk. They are not alternatives — they are the same machine at two time scales.

Transport Success Is Not Application Success

The most insidious lines in the August 23 log were not the 402 or the 429. They were these: mistral returned HTTP 200 — with truncated JSON. nemotron returned HTTP 200 — with prose mixed into the schema. If your monitoring watches HTTP codes, both requests were "successful." They were not. Transport success and application success are different things, and the essence of LLM observability lives exactly in that gap.

That is why output validation is not one layer but three:

1. Schema validation: does the JSON parse, are the fields complete, are the types right?
2. Semantic validation: do the values obey the business rules? Is score within 0–10, is decision consistent with score, is the justification actually grounded in the source text? A model can talk nonsense inside flawless JSON — a syntax guarantee is not a verdict-quality guarantee.
3. Consistency validation: do verdicts for the same input stay within an acceptable band over time?

And when the schema layer fails, resending the identical prompt is wasteful. The more deterministic move is a repair attempt: the second try includes the validation failure itself — "the previous output failed validation: field score missing, decision must be an enum; return JSON only." If the repair also fails, switch models. But there is a line: the moment you start performing regex surgery on broken JSON, you are not writing a fix — you are writing a new error class. That road leads to Frankenstein.

Observability: A Router Cannot Fly Blind

The real cost of the August 23 night was not the missing article; it was the outage staying invisible until morning. Because the fallback chain swallowed every error and went quiet with "the next run will try," the fact that six paths had died at once never turned into an alert. This is the structural risk of fallback architectures: the longer the chain, the more silent individual failures become — until all links die and the silence becomes a full outage.

A production-grade router's monitoring set looks roughly like this:

Two of these are worth more than the rest. The schema failure rate was the one metric that would have made our HTTP-200-but-broken outputs visible — I wish we had been watching it that night. Step depth is the cheapest early warning: within fallback execution, Cloudflare AI Gateway provides it out of the box via the cf-aig-step response header (0 primary, 1 first fallback); in your own router you carry the same information as a log field. If average step depth drifts away from zero, your primary capacity is rotting before users notice. And the chain exhaustion counter above zero means, by definition, "the user got an error" — the most serious event a router can produce. Without measurement a router is a black box, and black boxes spend their worst nights telling no one.

The Managed Alternative: Delegating to a Gateway

For anyone who does not want to carry all this machinery in their own code, managed gateways exist — with one currency check needed. Cloudflare AI Gateway used to offer fallback chains via the Universal Endpoint, where an ordered provider array was written into the request body. That endpoint still works for existing integrations but is now deprecated; for new integrations Cloudflare recommends the OpenAI-compatible REST endpoints, and for fallback/retry/conditional routing, Dynamic Routing. Dynamic Routing is a flow built through a visual interface or JSON configuration: conditional nodes, percentage splits (A/B and gradual rollouts), rate and budget quotas, model nodes, and fallback branches — in other words, a managed version of the machine described in this post. Retry and timeout behavior can also be delegated to the gateway via request headers (cf-aig-max-attempts, cf-aig-retry-delay, cf-aig-backoff, cf-aig-request-timeout).

The price of this approach is loss of control: you use the error taxonomy and the state machine only to the extent the gateway offers them. For small teams that is usually the right trade; if you need custom behavior per error class (like our reset-stamped quota quarantine), your own router layer becomes unavoidable.

A Design Checklist

The checklist distilled from the outage and the design of the tools above:

1. Count failure domains, not providers. Three API keys bound to the same project are one quota; two paths landing on the same inference backend are one path. Keep at least one path with paid, predictable quota and capacity on the critical path — and do not confuse "paid" with "under SLA": if you need a contractual SLA, verify it separately.
2. Do not write fallbacks before classifying errors. Retrying a 402, resending a context overflow to the same model — these produce noise, not resilience. And base the classification on the quadruple of status + provider error code + body + history, not on the HTTP code alone.
3. Consolidate retries into one layer and budget them. Multi-layer retries without jitter and without a budget reenact the SRE book's 64-request example, LLM edition.
4. Build the circuit breaker as a state machine. HEALTHY → OPEN → HALF_OPEN; reset-at stamps for quota, probe conditions for auth. A "permanent blacklist" is not a state machine — it is a pre-order for a future outage.
5. Measure application success, not transport success. HTTP 200 plus broken JSON is not success; make the schema failure rate a first-class metric.
6. Make retries idempotent. Carry a request_id end to end; separate inference retries from business operation retries.
7. No silent waiting. When the chain runs empty, the error must propagate up and reach an alarm — a system that goes quiet with "the next cron will try" hides an outage for hours.

Conclusion

A multi-LLM router is a genuine resilience layer when built correctly; built incorrectly, it is a complexity tax that multiplies your failure surface. The difference crystallizes in a handful of decisions: computing failure domains separately from provider counts, fallback chains separated by error class, idempotent retries budgeted in a single layer, and a circuit breaker built as a state machine.

But I saved the most important distinction for last. Most of this post answered "what to do when a request fails" — and that is not a good router's main job. Its main job is never sending the request to a deployment that is likely to fail in the first place — recomputing "who is eligible right now?" on every request from health state, quota scope, and latency percentiles. Fallback is the second line of defense. Ask your own setup this question: "what does my system do if all my providers die on the same night?" — if the answer is "it waits silently," your router is not finished yet.

Official Sources

• LiteLLM — Router: load balancing, fallbacks, retries (GitHub)
• Cloudflare AI Gateway — Dynamic Routing
• Cloudflare AI Gateway — Request handling: timeouts and retries
• Google SRE Book — Addressing Cascading Failures