My Agent Spent $23 Overnight. Here's Why My Limits Held and My Alerts Didn't.

I woke up to a notification that said my nightly agent had failed. That's the good version of a bad morning — something went wrong, and I knew about it.

Then I opened the bill. Twenty-three dollars on a cloud model, spent between roughly 01:40 and 04:10, on a job that normally costs about forty cents. So the thing I actually want to write about isn't the twenty-three dollars. It's the timeline, because the interesting part is what happened around 01:40.

The cap fired. The job stopped. The log has the entry. And I found out five hours later, from a different job that failed for an entirely separate reason.

What the job does, briefly

Every night at 01:30, a scheduled job reads my notes, pulls a few feeds, and writes a draft summary to a local file. Ninety-five percent of the work runs on a local model. A cloud call is there for one narrow escalation: if the local model reports low confidence on a section, the job re-runs just that section on a bigger model via API.

That escalation is supposed to fire a handful of times a night. On the night in question it fired continuously for two and a half hours, because of a retry loop I wrote, tested in the happy path, and never once tested with a model that kept saying "low confidence."

Each retry was cheap. The loop wasn't. That's the whole shape of the bug: nothing in it is expensive, and the product of it is.

The part I got right, mostly by accident

The spend cap held. I put a daily budget ceiling in the executor months ago, after I wrote up the guardrail stack I use before anything touches real money. Same three-number pattern: a per-run cap, a per-day cap, and a freeze when the day's cap trips.

The reason it held is a design choice I'd make again: the cap is mechanical, not statistical. It doesn't ask a model whether this call is reasonable. It doesn't consult a moving average. It checks a counter, and when the counter crosses a line, calls stop being made.

class Budget:
    """Spend caps. No model, no heuristics. A counter and a lock."""

    def __init__(self, per_run_usd, per_day_usd, store):
        self.per_run = per_run_usd
        self.per_day = per_day_usd
        self.store = store  # survives restarts; this is the whole trick

    def authorize(self, estimated_usd):
        spent_today = self.store.get("spent_today")   # from the ledger, not memory
        run_spent = self.store.get("run_spent")

        if run_spent + estimated_usd > self.per_run:
            raise Denied("per-run budget exceeded")
        if spent_today + estimated_usd > self.per_day:
            raise Denied("daily budget exceeded — freezing")

        self.store.incr("spent_today", estimated_usd)
        self.store.incr("run_spent", estimated_usd)

    def settle(self, actual_usd, estimated_usd):
        self.store.incr("spent_today", actual_usd - estimated_usd)
        self.store.incr("run_spent", actual_usd - estimated_usd)

Two details in there did all the work:

The counter lives in a store, not in a process variable. The first version of my cap kept spent_today in memory. Any restart — and my agent restarts a lot — reset the counter to zero. A cap that resets when the thing it's capping restarts is a cap made of wishes. Moving it to a small on-disk ledger was a four-line change and it's the difference between the cap working and the cap being theater.

Authorize before the call, settle after it. I reserve against the estimate, then correct to the actual. If the actual comes back bigger than the estimate, the ledger absorbs the difference — which means the next call is judged against reality, not against my optimism. My estimates were reliably low. That's not a bug I fixed, it's a bias I compensated for.

The cap is not the alert

Here's the mistake, and it's the reason this post exists.

When the daily cap tripped, my code did the correct thing: it raised Denied("daily budget exceeded — freezing"), wrote an audit entry, and returned. Clean, fail-closed, no drama.

And it logged that at INFO.

My alerting rules only looked at ERROR and above, plus a couple of specific failure classes I'd enumerated. So from the alerting system's point of view, the night was perfect. No error. Nothing to see. The one job that did notify me was the morning job, and it failed because the draft file it expected never got written — a downstream symptom that arrived five hours after the causal event.

The lesson generalizes past my stack: a fail-closed cap is a refusal, and refusals are not exceptions. From inside the system, refusing is the system working. From outside, refusing is the system not working. Only a human can tell you which — and the human needs to be told at the moment it happens.

The fix was one line of category, plus one line of routing:

# this was the bug, dressed as good hygiene
log.info("daily budget exceeded — freezing")

# what it is now: a refusal is an incident, not a status
log.warning("BUDGET_FROZEN", extra={"spent": spent_today, "cap": self.per_day})

And BUDGET_FROZEN is now on the alert list — not the page-me-now list, but the tell-me-immediately list. There's a real distinction there. A budget freeze at 01:40 doesn't mean wake the human; it means the pipeline is dead until morning, and the human gets to know before the morning job fails, with the number attached. Twenty-three dollars is a shrug. Twenty-three dollars discovered at 04:15 by a confused downstream job is a five-hour blind spot.

The second bug: the loop had no shape

The retry loop is a separate story and it's more embarrassing, so here it is quickly.

My escalation logic was: if confidence is low, retry on the bigger model. I tested it by mocking a model that returns "low confidence" once and then confidently. So in my head, the loop was a single hop.

In reality, the bigger model also said low confidence, because the section genuinely was ambiguous, and the section never got less ambiguous, because I never changed anything between attempts. It retried the same query against the same input with the same context and hoped for a different answer. That is, definitionally, the thing I have a circuit breaker for — and the breaker didn't catch it, because every individual call succeeded. A 200 is a success. The breaker watches for failure, and my loop produced nothing but successes until the money ran out.

So the retry loop got two additions:

• Attempt cap. Three tries, hard stop. Not "until it works."
• Escalation must change the input. Each retry has to add something — more context, a decomposition step, a different prompt. A retry that repeats the previous attempt byte-for-byte isn't a retry, it's a subscription.

And the effect-count check I now put on every job — the assertion that a run must prove it produced something — is exactly what would have caught this at 01:45 instead of 04:10. The job was producing effects, so it would have passed. But the loop had no cost assertion, and cost is an effect too if you're paying for it by the token.

Two metrics that actually predict this

Post-incident, I stopped watching "spend total" as my primary number and started watching two ratios:

Spend per unit of work. Dollars per draft, per filing, per summary. This is the number that would have been screaming. My absolute spend looked fine for the first hour — forty cents an hour is normal-ish. It was spend per draft that went vertical, because the job was producing zero drafts while burning tokens. Same shape as the flatline I wrote about in the silent-failure post: the aggregate looked normal, the ratio was on fire.

Retry ratio: retries divided by first attempts. Mine sits near 0.05 on a normal night. Anything above ~0.3 means a loop or a degraded model, and it's visible in minutes rather than dollars. Cheap to compute, no new infrastructure, and it's the only number that moved before the bill did.

One more, and it's the uncomfortable one: the alert you're not watching is the alert you've muted. I'd been getting budget-adjacent notices for weeks and had stopped reading them, because most were routine. That's not an alerting failure, it's a calibration failure — and it's the same terminal state I wrote about with approval gates that ask too often. Any channel that asks more than a few times a month stops being oversight. If you're drowning in routine notices, the fix is fewer notices, not more attention.

What I'd tell you to copy

If you run anything that spends money on your behalf — cloud LLM calls, API credits, compute — here's the short version:

1. Caps in the executor, not the model. A counter and a line. No model gets a vote on whether to spend.
2. Counters survive restarts. If it lives in memory, it isn't a cap.
3. Reserve, then settle. Judge the next call against the real number, not your estimate.
4. A refusal is an incident. Log it at a level your alerts can see, with the number in the message.
5. Cap your retries, and make each retry change something. A repeated call isn't a retry.
6. Watch the ratio, not the total. Spend per unit of work is the early-warning number; the total is the receipt.

The cap worked exactly as designed. The $23 was the cap doing its job — the difference between a runaway bill and a bounded one. The failure was that I didn't hear it close. A guardrail you don't get alerted about is a guardrail you're trusting on faith, and I'd rather have the boring version: it stops, and it tells me it stopped.

Curious how other people handle this — do you alert on budget refusals immediately, or roll them into a digest so you don't train yourself to ignore them? I keep going back and forth. Drop a comment with what's worked.

Part of my Building in Public series — previously: the assertion that catches runs that do nothing, timeout means no on approval gates, the guardrail stack before going live, and a circuit breaker that caught three outages.