On a Friday afternoon a colleague added a label to a counter so he could follow one customer's orders through a queue. The label was the order id. It went through review in four minutes, because it was three characters of change and the reasoning behind it was good.
By Sunday evening our metrics server was restarting in a loop, and on Monday morning, in the middle of a genuine payments incident, we had no monitoring at all.
A label is not a field on a log line. It is a dimension, and every distinct combination of label values is its own time series, with its own entry in a memory resident index and its own block of samples. Our order ids are unique, sixty thousand of them a day, on a counter that four services increment. Those series do not go away when the order does, because they have to be queryable for as long as the retention window. We went from around one point one million active series to fourteen million in under three days.
The failure was not clean. Ingestion slowed first, which delayed rule evaluation, which meant alerts arrived minutes late and some evaluation cycles were skipped altogether. Then the process was killed for memory, came back, spent twenty minutes replaying its write ahead log, and was killed again. The system we would normally use to understand an outage was the system having one.
We dropped the label at the scrape with a relabel rule before touching the application, because that stops it without waiting on a deploy. Then the guards went in. Every service has a series budget and a rule that fires at eighty percent of it. A check in CI rejects any metric whose label name ends in id, or matches a short list of words including email, path and url. A panel shows the top metric names by series count, which none of us had ever looked at and which made the cause obvious in about ten seconds.
The identifier itself was a reasonable thing to want. It belongs on a trace, or on an exemplar attached to a bucket, where it costs one record rather than a whole dimension multiplied across every dimension you already had.
– Sergey Shinder
