A partner switched on a new storefront at nine on a Thursday morning and sent us about a fifth more traffic than a normal peak. Twenty minutes later our order API was returning nothing usable at all, on hardware that was not close to full, in a service that had absorbed bigger jumps than that before.
The shape of the failure is what took us so long. Nothing crashed. No instance restarted. Error rates stayed low for most of it, because the requests were not erroring, they were being answered late to callers who were no longer there.
In front of our sixty four workers sat an accept queue with no bound on it, which is the default in the framework we use and which none of us had ever set. Clients give up after two seconds and retry twice. Once arrivals crept past what the workers could clear, the queue grew, and queue wait went from a few milliseconds to six seconds and then nine. Every request a worker picked up had already been abandoned by its caller, who had already sent a replacement, which was queued behind everything else. We were running at full CPU, completing thousands of requests a minute, and delivering almost none of them to anyone who was still listening. Queue depth peaked a little over eleven thousand.
A timeout on the client side is a decision the client makes about its own patience. Our server knew nothing about it, so it spent the whole incident doing work that had already expired, and the retries that expiry generated were the thing keeping the queue full.
The queue is bounded now, sized from the worker count and the latency we intend to hold, and a full queue returns a refusal immediately rather than accepting work it cannot reach in time. Callers send a deadline, which workers check before starting and before each downstream call, and expired work is dropped with a counter that we graph. Our client library retries on a budget with jitter instead of a fixed count.
And our load tests go past the knee. We had always tested up to the rate we wanted to serve and stopped there, so we had measured how the service performs and never once how it fails.
– Sergey Shinder
