The disk alert on our primary database volume fired at 14:02. It was correctly written, correctly routed, and acknowledged by a human inside two minutes. At 14:19 the volume was full and the database stopped accepting writes.
Our procedure for this is to drop the oldest partitions, then extend the volume and grow the filesystem. Done calmly it takes half an hour. We had seventeen minutes, and we spent the first six of them establishing that the alert was real. Nobody did anything wrong that afternoon. The alert simply arrived far too late to be acted on, and it had been arriving too late for months without anyone noticing, because until that day it had never arrived at all.
The threshold was eighty five percent and it was correct when it was written. The volume was two hundred gigabytes then and we wrote about four gigabytes a day, so eighty five percent meant three or four days of warning. The volume is now four terabytes, and a retention change in June took us to roughly three hundred gigabytes a day. The threshold did not move. Nobody needed to move it. It kept meaning exactly what it said, and what it was worth in minutes quietly collapsed.
A percentage tells you where you are. It says nothing about how fast you are travelling, and the only number that is useful at 14:02 is how long you have.
Every capacity alert we own now fires on projected time to exhaustion rather than a level. A linear fit over the last six hours, alerting when the resource is projected to run out within four hours, with a second, louder one at one hour. The alert text reads projected full at 16:40, roughly two and a half hours, which is a sentence an on call engineer can make a decision from at three in the morning.
Then we audited the rest. Eleven capacity alerts, each one replayed against its own history to work out how much notice it would actually have given at the observed growth rate. Four gave less than fifteen minutes. The audit log volume gave ninety seconds. Two, on things that had not grown in years, gave several weeks and were fine.
Levels are easy to write and easy to agree on. Rates are the ones that tell you whether you are going to make it.
– Sergey Shinder
