There's a whole genre of posts about waking up to a surprise OpenAI bill.
They all end with the same advice: set a budget cap. Sensible. What nobody
talks about is how those caps actually work under the hood, so I got
curious and started reading implementations. Tutorials, production
snippets, open source tools, whatever I could find.
Almost all of them share the same bug. Not a typo kind of bug, a
shape-of-the-problem kind of bug. This post is about what it is, how bad it
gets (I measured), and the small open source tool I ended up building
because I couldn't leave it alone.
The bug
Give your AI agent a $50/day budget and ask yourself: can it spend $63?
With most setups, yes. Here's the pattern I keep seeing, in tutorials, in
production code, in tools that should know better:
- read the spend counter
- if there's room under the cap, make the LLM call
- when the response arrives, add its cost to the counter
The problem lives between step 1 and step 3. An LLM call's cost doesn't
exist until the response comes back. So while one call is in flight, ten
other requests are reading a counter that knows nothing about it. All ten
pass the check. Your cap was $50 and you're at $63.
Normal rate limits don't have this problem. A request costs "one request"
and you know that up front. Spend is different: the thing you're limiting is
only measurable after the fact, so the counter is always behind reality, and
it falls further behind exactly when traffic spikes. Which is when you
wanted the cap most.
To be fair, the popular proxies have done real work here. LiteLLM moved its
rate limiting onto atomic Redis Lua scripts back in v1.72.2, which killed
the read-check-write race inside a single check. But spend tracking is a
different subsystem: cost gets computed after the response and written
asynchronously (batched writers, cached counters), and nothing holds the
estimated cost of in-flight calls against the budget. I'm not picking on
anyone. It's a genuinely awkward shape of problem.
Measuring it instead of arguing about it
I wrote a demo that runs two limiters under the same load: 1,000 concurrent
requests against a cap of 500. One is naive check-then-act against a Redis
counter. The other does the check and the deduction inside one Lua script.
cap = 500, concurrent requests = 1000
naive check-then-act: granted 613 (over-spend: +113)
atomic enforcement: granted 500 (over-spend: +0)
I ran it four times. The naive limiter over-spent by +113, +124, +148 and
+161. A different number every run, because it's a race, and races don't
owe you consistency. The atomic one granted exactly 500 every time.
That's a 20-30% overshoot on a "hard" cap, from nothing but concurrency.
The fix has two parts
The first part is obvious once you see it: the check and the deduction have
to be one operation. Redis executes a Lua script atomically, nothing
interleaves, so the race window just doesn't exist:
-- abridged: all-or-nothing across every configured period
local allowed = true
for _, b in ipairs(budgets) do
if b.remaining < amount then allowed = false end
end
if allowed then
for _, b in ipairs(budgets) do
redis.call('DECRBY', b.remaining_key, amount)
end
end
The second part took me longer. Atomic or not, you can't deduct a number
you don't know yet, and at decision time the LLM call hasn't happened.
Payment systems solved this decades ago with two phases, so I stole that:
- reserve happens before the call: atomically hold an estimate against the budget. If the daily cap or the monthly cap would be blown (they're checked together, all-or-nothing), the request is rejected and nothing is held.
- settle happens after the response: report what the call actually cost. The difference gets refunded. If the actual ran over your estimate, the difference gets charged, and yes, that means a budget can end up slightly negative.
I went back and forth on that last part. But a cap on unknown costs can be
strict at admission time or it can be clairvoyant, and it can't be both.
I'd rather the system admit that than pretend.
Crashes were the other design headache. If a client reserves and then dies,
that hold can't sit there eating budget forever. So every reservation
carries a TTL and gets auto-released. There's no background worker for
this; expired holds get reclaimed lazily on the next call for that subject,
in bounded batches, which keeps enforcement latency flat.
Does it hold up
The test I care most about fires 1,000 concurrent 1-unit requests at a
500-unit budget and asserts that exactly 500 succeed. Not "at most 500".
Exactly. There's an equivalent for reserve/settle that checks the books
reconcile to the unit after 200 concurrent reservations with a mix of
settles and cancellations.
On my laptop it does about 9,000 enforcement decisions per second with p99
around 5ms, and a multiprocess drain test shows zero over-serving.
What it doesn't do
- Rate limiting. The proxies already do that well, atomically.
- Redis Cluster. Single instance for now.
- Estimate your costs for you. Estimates are your job; reserve/settle just makes a wrong estimate safe instead of free.
Try "Hardcap"
The whole thing is a small FastAPI + Redis sidecar called hardcap,
Apache-2.0: https://github.com/pradyumnvermaa/hardcap
git clone https://github.com/pradyumnvermaa/hardcap && cd hardcap
docker compose up -d && python demo/overspend_demo.py
The demo prints the naive limiter's over-spend on your machine. It will be
a different number than mine. That's kind of the whole point.
And if you look at the reserve/settle semantics and think you can make it
over-spend, I genuinely want to hear about it.
