Your retry loop is your most expensive feature

A team I worked with tracked cost per task down to the token. They capped context. They set a budget alert. And then they added retries: 3 to the model client and never put it in the budget.

That retry path only fires when the model is slow or down — i.e. at peak load. And every retry replays the entire context. So the one part of the system nobody accounts for is the part that costs the most, exactly when you can afford it least.

The number that surprises people

Take a support agent for an ASEAN e-commerce shop. A hard ticket pulls a 50k-token context (order history, policy docs, prior turns) and emits ~1k tokens.

• One attempt ≈ 51k tokens.
• With retries: 3 and the task still failing, you pay for 4 attempts: 51k × 4 ≈ 204k tokens for a task you budgeted at 51k.

That's a 4× tax — and it isn't spread evenly. The tasks that fail are the expensive ones: long context, heavy tool chains, messy inputs. Retries concentrate exactly where cost is highest.

retry_policy:
  naive:
    attempts: 3
    on_failure: "replay full context to same model"
    hidden_cost_multiplier: 4        # 1 attempt + 3 full-context replays
    shows_up_in_dashboard: false
  cost_aware:
    attempts: 3
    on_failure: "route to next equivalent model"
    carries_context: true
    hidden_cost_multiplier: 1        # context travels, never replayed
    shows_up_in_dashboard: true

Why the bill stays invisible

Most cost dashboards report average cost per successful request. A task that fails three times and finally succeeds doesn't appear as "4× cost" — it appears as one success with a latency blip. The retry is a hidden fourth axis of spend:

• Latency triples → someone notices.
• Cost triples → almost nobody measures it, because it's buried in "failed then retried" traffic.

So teams tune the happy path and ignore the one code path that runs under stress and multiplies the bill.

The fix is routing, not retrying

You don't make retries cheaper by retrying harder. You stop replaying context: when a model is unavailable, advance the task to the next equivalent model that can take the same handoff context. The context travels with the task; you don't re-pay for it as a "retry."

With 25+ models behind one OpenAI-compatible endpoint, "equivalent" stops being theoretical — there's a tier you can actually route to without rewriting prompts or adapters. For teams under data-residency rules (PDPA-aligned, SG-hosted), this also matters: re-running context on a different provider just to "retry" raises compliance questions, not just cost.

flowchart LR
  subgraph Naive[Naive retry: context replayed]
    N1[Task · 50k ctx] --> N2[Attempt 1: 50k in]
    N2 -->|fail| N3[Replay 50k in]
    N3 -->|fail| N4[Replay 50k in]
    N4 -->|fail| N5[Replay 50k in]
    N1 -.paid 4x.-.-> N5
  end
  subgraph Smart[Cost-aware: context travels]
    S1[Task · 50k ctx] --> S2[Model A call]
    S2 -->|unavailable| S3[Model B · same ctx]
    S3 -->|unavailable| S4[Model C · same ctx]
    S1 -.ctx paid once.-.-> S4
  end

def step(task, model_registry):
    # cost-aware: context is paid ONCE, then travels with the task
    for model in model_registry.ordered_for(task.capability):
        try:
            return model.call(task.handoff_context)   # no replay
        except Unavailable:
            continue                                   # next equivalent, same context
    raise NoCapableModel

# for contrast — the naive version re-sends the FULL context every attempt
def naive_retry(task, model, attempts=3):
    for _ in range(attempts):
        try:
            return model.call(task.full_context)        # full context RE-SENT each time
        except (Timeout, Unavailable):
            continue
    raise RetryExhausted

The router doesn't retry the model. It advances the task to the next model that can take its context. Same uptime, a fraction of the bill.

The question I'll leave you with

Before you ship retries: 3: for the tasks that fail, how many tokens does the retry actually replay — and is that number in any dashboard you look at?

No tool to sell, no link to click. Just this: most LLM systems that survive a model outage are just retry loops with better logging — and the loop is the most expensive thing they do.