Postmortem: The Retry Storm That Ate a Month of Quota

Saturday morning, the weekly digest table had 900 rows instead of 3,000, and the dashboard was a cheerful green. The job had exited with code 0, the dead-letter file had grown to twelve megabytes, and nobody had opened it in three weeks. The only thing that actually noticed the failure was a quota email that arrived forty minutes after the damage was done.

You have probably built this job yourself: fan out over a few thousand items, ask a model for a two-sentence summary of each, write the rows, exit. Nobody watches it because it has worked for months, and the only signal wired to a human is the exit code. That signal becomes a lie the moment your retry loop lives inside a try block that ends with continue.

The timeline below is a reproduction you can rebuild in about twenty minutes against a local stub, which is the point: a retry storm is a design defect, not bad luck, and it should be cheap to prove.

Timeline, reconstructed

T+0 — eight workers start on a 3,000-item queue with one API key and no shared spend counter. Each worker believes its own retry budget is the global budget.

T+6m — the endpoint begins answering 429. The backoff is 2 ** attempt with no jitter and no cap, so every worker sleeps the same amount and wakes up together, which turns a rate limit into a synchronized stampede.

T+22m — the retry counter resets for each new item, so a single item can burn six attempts while the run never trips any limit of its own. Throughput looks healthy on the graph because the graph counts requests, not rows.

T+41m — the quota is gone. The provider now returns a hard quota error, the classifier treats it as retryable because it is still an HTTP error, and the storm keeps going at full speed against a wall.

T+6h — the run finishes exit 0 with 71% of rows missing. Nothing paged, because nothing was ever wired to page: failed items were appended to a file, and files do not wake anyone up.

Three contributing factors, one root cause

The classifier is the first factor. Treating every non-200 response as retryable is the same mistake as treating every exception as transient; 401, 403, 404, and hard quota errors are terminal, and retrying them only spends money to confirm the obvious.

The second factor is the missing ledger. Nobody could answer the only question that mattered at T+41m, which was how many tokens this run had consumed so far, because that number existed only inside a provider dashboard that refreshed hourly.

The third factor is the silent dead letter. A failed item that lands in a file with no count, no age, and no alert is not handled; it is hidden. The root cause underneath all three is that the job had no notion of a run-level budget that both code and humans could read.

The durable fix: a ledger, a cap, and a softer lane

The fix is a token ledger you can audit, written before you need it. One row per attempt, keyed by run, with usage from the response rather than an estimate from a tokenizer you did not pin.

# guard.py — a token ledger you can query at 2am
import random, sqlite3, time

DDL = """
CREATE TABLE IF NOT EXISTS attempts (
  id      INTEGER PRIMARY KEY AUTOINCREMENT,
  run_id  TEXT    NOT NULL,
  item_id TEXT    NOT NULL,
  attempt INTEGER NOT NULL,
  model   TEXT    NOT NULL,
  ptok    INTEGER NOT NULL DEFAULT 0,
  ctok    INTEGER NOT NULL DEFAULT 0,
  status  TEXT    NOT NULL,
  err     TEXT,
  ts      REAL    NOT NULL
);
"""

RETRYABLE      = {"429", "500", "502", "503", "504"}
BUDGET         = 200_000   # tokens per run, deliberately below the monthly ceiling
MAX_ATTEMPTS   = 4

class BudgetExhausted(RuntimeError):
    pass

def open_ledger(path="ledger.sqlite"):
    conn = sqlite3.connect(path, isolation_level=None)
    conn.execute(DDL)
    return conn

def spent(conn, run_id):
    q = "SELECT COALESCE(SUM(ptok + ctok), 0) FROM attempts WHERE run_id = ?"
    return conn.execute(q, (run_id,)).fetchone()[0]

def log(conn, run_id, item_id, attempt, model, ptok, ctok, status, err=None):
    conn.execute(
        "INSERT INTO attempts (run_id,item_id,attempt,model,ptok,ctok,status,err,ts)"
        " VALUES (?,?,?,?,?,?,?,?,?)",
        (run_id, item_id, attempt, model, ptok, ctok, status, err, time.time()),
    )

The call site then refuses to start work it cannot pay for, and it separates "wait" from "stop" by using status codes instead of exception types.

def summarize(client, conn, run_id, item, model="free-lane"):
    for attempt in range(1, MAX_ATTEMPTS + 1):
        if spent(conn, run_id) >= BUDGET:
            log(conn, run_id, item.id, attempt, model, 0, 0, "budget_stop")
            raise BudgetExhausted(run_id)
        try:
            r = client.chat.completions.create(
                model=model,
                max_tokens=160,
                messages=[{"role": "user", "content": item.prompt}],
            )
            u = r.usage
            log(conn, run_id, item.id, attempt, model,
                u.prompt_tokens, u.completion_tokens, "ok")
            return r.choices[0].message.content
        except Exception as e:  # narrow this to your SDK's error base
            code = str(getattr(e, "status_code", None) or type(e).__name__)
            log(conn, run_id, item.id, attempt, model, 0, 0, "error", code)
            if code not in RETRYABLE or attempt == MAX_ATTEMPTS:
                raise
            time.sleep(min(2 ** attempt, 20) + random.uniform(0, 1))

Jitter is not a nicety here; it is what stops eight workers from becoming one very expensive worker. The cap at twenty seconds keeps a long tail from stretching a nightly run into a morning run.

A preflight check costs one request and prevents the worst version of this incident, where a batch starts with an already-dead key.

curl -sS -o /dev/null -w '%{http_code}\n' \
  "$BASE_URL/models" -H "Authorization: Bearer $KEY"

Finally, split the lanes. The summary job stays on whatever model you trust for quality, and a second, cheaper lane handles verification: re-running a small sample of summaries, checking that the JSON wrapper still parses, and confirming that the row count reconciles against the queue. That lane is where a free model endpoint genuinely fits, because its job is to be suspicious, not authoritative.

This is also where MonkeyCode enters the picture, and the fit is narrow enough to describe honestly. The project is open source, and its materials advertise free model access up to roughly 10 million tokens plus a free server option, which makes it a reasonable home for exactly that low-stakes verification lane — capacity, limits, and duration are set by the project, so read the current terms before you depend on them for anything scheduled. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The test plan that proves the guard works

You cannot verify a retry guard against a healthy endpoint, so build a fault injector. Forty lines of standard library will do, and it never costs a token.

# stub.py — a 429 machine, so you can watch the guard behave
from http.server import BaseHTTPRequestHandler, HTTPServer

class H(BaseHTTPRequestHandler):
    def do_POST(self):
        body = b'{"error":{"message":"rate limited","type":"rate_limit_error"}}'
        self.send_response(429)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)
    def log_message(self, *args):
        pass

HTTPServer(("127.0.0.1", 8787), H).serve_forever()

Point BASE_URL at http://127.0.0.1:8787 and run a twenty-item fixture. Then assert four things rather than eyeballing logs: total attempt rows never exceed twenty times MAX_ATTEMPTS, the process exits non-zero instead of zero, the dead-letter count equals twenty, and your alert fires. Flip the stub to fail thirty percent of requests and assert that successful rows plus dead letters still equal twenty, which is the check that would have caught the original incident in a single line.

After a real run, the ledger answers the audit question directly.

SELECT status, COUNT(*) AS n, SUM(ptok + ctok) AS tokens
FROM attempts WHERE run_id = ? GROUP BY status;

Limitations and who should skip this

The ledger is per-run and per-file, so if you scale to many machines you need a real database and a row lock, not SQLite on a shared volume. Streamed responses may not report usage the same way across providers, which means your ptok/ctok columns can drift from the invoice and should be treated as an upper-bound estimate rather than accounting. Free tiers change without notice, so never let a verification lane be the only thing standing between you and a silent data gap.

Skip this approach entirely if you need contractual uptime or data residency guarantees, if the workload is latency-sensitive, or if your volume is large enough that the engineering time in this guard costs more than the tokens it saves. It is also the wrong pattern for regulated data you have not cleared for a third-party endpoint, no matter how free the endpoint is.

What you should keep, regardless of provider, is the habit of asking the run itself how much it has spent. If you want a zero-cost place to rehearse the verification lane, the project's free tier is a reasonable sandbox — just keep your own ledger, and let the stub, not your quota email, be the thing that tells you the truth.