The HTTP 429 That Turned Seven Minutes Into Zero Work

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

ARGUS is a data-catalog governance swarm. Its agents use structured model calls to draft descriptions, classify sensitive fields, and review repairs before anything can be written.

During a 45-entity sweep, the system spent seven minutes producing zero proposals. The report called the result review failed.

The endpoint was reachable. Health probes returned 200. The logs showed only:

429 Too Many Requests

The retry loop was working exactly as written. That was the bug.

Two different failures shared one status code

The model provider enforced both a short burst limit and a daily token quota. Both failures arrived as HTTP 429 with the same useful headers.

Their remedies are opposites:

The only reliable distinction was in the response body. The provider's daily-quota response explained:

on tokens per day (TPD): Limit 100000, Used 99299, Requested 1139

ARGUS never logged that text. The OpenAI-compatible client raised an httpx status exception whose default message contained the status and URL, but not the response body.

The information needed to fix the failure was present on the wire and discarded at the error boundary.

Why the health probe lied

The account was close to its cap, not necessarily at exactly zero remaining tokens. A tiny probe could still fit under the allowance and return 200.

The real structured request included a schema, column context, lineage context, and reserved output. It needed more than the remaining budget and failed.

That created a misleading picture:

• connectivity looked healthy
• authentication looked healthy
• small calls looked healthy
• every useful call failed

A probe only proves that the probe can run. It does not prove that the real workload fits inside the remaining quota.

The retry loop amplified the outage

ARGUS treated every 429 as temporary. Each model call could wait through six backoffs. A sweep could need roughly one hundred calls.

For a rate limit that clears in seconds, this is patient behavior. For a daily quota that resets tomorrow, it turns one deterministic failure into hours of waiting.

The retry policy had no concept of recoverability. It knew that a request was throttled, but not whether time inside the current operation could change the result.

Fix one: preserve the provider's explanation

The HTTP boundary now reads and carries a bounded copy of the response body when it raises an error.

Instead of this:

429 Too Many Requests

the log can say which limit was hit, how much was used, and what the failed request needed.

That single change identified the root cause on the next run.

The broader lesson is simple: an error abstraction should hide irrelevant transport details, but it must not erase the only field that distinguishes two operational states.

Fix two: model exhaustion as its own outcome

I added a BudgetExhausted exception rather than routing a daily quota through the ordinary retry path.

When the response identifies a long-lived token or request quota:

1. The call fails immediately.
2. The endpoint is retired for the current sweep.
3. A breaker prevents the remaining agents from repeating the same doomed call.
4. The breaker resets at the start of the next sweep.
5. The CLI labels the totals as partial.

The reset matters. A process-wide flag that survives its cause would make tomorrow's healthy quota look permanently dead.

Fix three: do not turn an outage into a verdict

The most important change was not in the HTTP client. It was in the Arbiter.

ARGUS already had a deliberate no-model mode. If the operator starts a run without a key, mechanical grounding becomes the declared review gate. A proposal that passes that gate can be approved.

Budget exhaustion halfway through a modeled run is different. Falling into the same branch would silently remove the review gate at the exact moment it stopped working.

The new outcome is:

not reviewed: model budget exhausted

It is not rejected, because no reviewer made a judgment. It is not approved, because the expected review never happened. Unreviewed proposals are never written.

This distinction protects both the data catalog and the truthfulness of the report.

Fix four: pace and rotate

The provider reports remaining tokens and reset timing on successful responses. ARGUS now uses those headers to pace before hitting the wall instead of discovering the wall by repeatedly crashing into it.

I also reduced reserved output from 4096 tokens to 900. The structured responses are short descriptions, tag lists, and verdicts. On free tiers, reserving 4096 tokens can consume rate-limit capacity even when the reply uses far less.

When multiple endpoints are configured, the retry and rotation loops are now separate:

• retry when the same endpoint may answer next time
• rotate when the endpoint will not answer during this sweep
• do not rotate on unrelated errors such as malformed output or HTTP 500
• trip the breaker only when every configured endpoint is unavailable

That last negative test matters. Rotating on every error would burn the entire provider chain because of one bad prompt.

Duration strings were another hidden bug

The original reset parser stripped a trailing s and called float().

Real providers return values such as:

1h30m43.2s
205ms

Both values failed the old parser and silently fell back to guessed delays. The replacement parser handles hours, minutes, seconds, milliseconds, decimals, and compound durations.

Before and after

The real exhausted endpoint that previously took seven minutes now produced a clear partial-run report in seconds and committed nothing.

The regression suite grew dedicated coverage for:

• daily quota versus temporary rate limit
• no retry after confirmed exhaustion
• breaker reset between sweeps
• no automatic approval after mid-run exhaustion
• unreviewed proposals never reaching a write
• duration parsing for 1h30m43.2s and 205ms
• rotation after persistent unnamed throttling
• no rotation on non-throttle failures
• stopping clearly when every endpoint is spent

What I learned

Status codes are categories, not diagnoses

HTTP 429 says the request cannot run now. It does not say whether waiting inside the current job can help.

Observability must preserve decision-making evidence

Logging more bytes is not automatically useful. Logging the one bounded provider field that distinguishes retryable from terminal failure is.

Degraded mode and outage mode are different contracts

Starting without a reviewer is an explicit operating mode. Losing the reviewer halfway through is a failed assumption. They should not share a fallback branch merely because both lack a model response.

A partial success must call itself partial

The sweep produced real numbers for fewer entities than requested. Presenting those totals without qualification would overstate coverage even if every individual number were correct.

The best part of this fix is that it did not merely reduce latency. It stopped an infrastructure outage from being misreported as an AI judgment and preserved the review boundary when the budget disappeared.

AI assistance was used during code and draft iteration. Joshua Hernandez is the sole contest entrant and is responsible for the submitted implementation and evidence.