The Agent Added a Retry Loop and Retried on the Auth Failure Forever

A service we depend on has been flaky for months. It returns 502s roughly one request in twenty, and when it does, retrying once usually fixes it. We asked the agent to add a small retry wrapper around the client so we stop seeing sporadic errors in our logs.

The agent came back with a clean result:

async function callWithRetry(path, opts, attempts = 3) {
  for (let i = 0; i < attempts; i++) {
    const res = await fetch(path, opts);
    if (res.ok) return res;
    await sleep(200 * (i + 1));
  }
  throw new Error(`call failed after ${attempts} attempts: ${path}`);
}

It looks reasonable. Three attempts, a short backoff, a clear error. We merged it. Tests pass.

Two days later, our token-refresh job starts failing. Then it stops even trying. Our error budget spikes, and a trace shows the same call being retried three times per invocation, then failing.

The underlying cause was boring: the service rotated its auth signing key, our token stopped being accepted, and the endpoint started returning 401 Unauthorized. The agent's retry loop does not care. A 401 is a non-2xx. It sleeps 200ms, retries, gets another 401, sleeps 400ms, retries, gets a third 401, and finally throws. Every single request now costs three round trips and one wasted backoff window, and the actual failure mode — "your credentials are bad" — is hidden behind a generic "call failed after 3 attempts" error.

This is the retry-loop trap. The agent treats "the call did not succeed" as a single condition worth retrying. In reality, HTTP failures split into three buckets that need completely different responses:

• Transient — 502, 503, 504, connection reset, DNS blip. Retry with backoff, maybe with jitter.
• Permanent client error — 400, 401, 403, 404, 422. The request is not going to succeed on a second try. Surface it immediately, alert on it, and let the caller fix the input or the credentials.
• Rate limited — 429. Retry, but honor Retry-After, do not roll your own backoff.

The agent's version lumps all three together. The diff is small, the tests are green, and the cost is invisible until a real permanent failure happens and turns into a small DDoS on your own caller.

A few things made this easy to miss:

• The tests mocked the client to return 502 on the first call and 200 on the second. They proved the retry works for the case we were trying to fix. They never exercised a 401.
• The agent's PR description said "added retry with backoff" — it did not say "retries on every non-2xx including auth failures."
• The error message that eventually surfaced — "call failed after 3 attempts" — discarded the original status code. By the time we read the log, the 401 was gone.

The cheap guard I have started asking for: before merging any new retry wrapper, make it classify the status code explicitly. A 401 or 403 should throw immediately with the original status preserved. A 429 should read Retry-After. Only 5xx and network errors get the backoff loop.

function shouldRetry(res) {
  if (res.status === 429) return true;
  if (res.status >= 500) return true;
  return false;
}

That one function is the difference between a retry loop that improves reliability and one that magnifies every permanent mistake. The agent will not write it on its own, because the spec it was given — "add retries for the flaky endpoint" — did not mention the auth case. The review question that catches this is simple: which status codes will this retry on, and is that list explicit in the code?

If the answer is "everything that is not ok," you have the bug before it has had a chance to wake you up at 3 a.m.