Through November our checkout failed in short bursts, about two seconds long, roughly every ten minutes. The five minute graphs barely moved, but customers saw every one. Redis latency spiked at the same moments, while its CPU averaged eleven percent.
SLOWLOG GET answered it in one line. Every ten minutes there was a KEYS cart:* that took between one and two seconds.
It came from a cleanup job, three years old, that finds abandoned carts and copies them to the marketing database before deleting them. Each cart is a key named cart: and a customer id. When the job was written we had a couple of hundred thousand of them and KEYS took a few milliseconds. By November, with guest carts kept for thirty days, the instance held around forty million keys.
Redis executes commands one at a time on its main thread. KEYS walks the entire keyspace, whatever the pattern, and replies only when it has finished, so for those two seconds every other command from every service waited. Our checkout's Redis client had a five hundred millisecond timeout, which was right for normal operation, and every request queued behind the scan timed out together. The Redis documentation is unusually direct: KEYS is for debugging, not for regular application code.
Nothing about the job had changed. Its cost was set by the size of the data, and the data had grown a hundred and fifty times while the job looked exactly the same in code review.
The job now iterates with SCAN and a COUNT of a thousand, which returns a little at a time and lets other commands run in between, and the abandoned cart list is kept as a sorted set by last update, so the job reads only the carts that are due. Application users have an ACL that denies the dangerous command category, so KEYS, FLUSHALL and their kind fail for anything but an operator. We alert on SLOWLOG length and on Redis latency at the ninety ninth percentile over one minute, not five. And the load test now runs against a copy with production's key count, not a few thousand keys made up for it.
A command that is fast on small data is not fast. On a server that does one thing at a time, its cost is paid by every client.
– Sergey Shinder
