At twenty past two on a Tuesday morning one customer began a historical backfill against our API. Within ten minutes a hundred and twelve other customers were being rejected, and the customer who started it was getting most of the throughput they had asked for.
Our edge holds a single token bucket for the whole service, two thousand requests a second, a number chosen years ago from what the service can actually serve. It is not per key. It never had been, and I had read the configuration many times without noticing that the absence of a key dimension was a decision about fairness rather than an implementation detail.
A shared bucket is drained by arrival. A client with nineteen hundred requests in flight takes tokens at nineteen hundred times the rate of a client with one, because every token that appears goes to whoever is asking at that instant, and asking is exactly what that client was doing. The limiter did its job perfectly. It kept the service standing, and it distributed the shortage in direct proportion to who was causing it, which is to say in inverse proportion to who deserved it.
That customer's steady rate is about forty a second. Our aggregate availability for the hour came out at ninety one percent, which reads as a moderate incident. For the hundred and twelve customers who were not doing anything unusual it was closer to thirty percent, and that number existed nowhere.
Each key now has its own bucket, sized from its own trailing thirty day peak with a multiplier, with the global bucket kept behind them as a backstop for the service rather than as the allocation mechanism. Requests carry a class, so an interactive call outranks a batch call from the same key. Responses say which limit was hit. And we record the throttled fraction per key, so shedding is something we can see happening to somebody specific.
The reporting changed too. We publish the worst tenant's success rate next to the aggregate, because the aggregate is an average over customers and had never once told us that a small number of them were having a completely different day.
A limit protects the service. It says nothing about who gets what, and where you have not said it, the answer is whoever pushes hardest.
– Sergey Shinder
