Checkout was returning errors for about one request in three and the dashboard in front of four engineers was entirely healthy. Request rate normal, error rate flat, latency at a hundred and eighty milliseconds. Everything on the screen was accurate, current and about a different environment.
When we consolidated onto one Grafana with a datasource per environment, every dashboard gained a datasource template variable. A template variable has a default, and unless you set one the default is whichever option sorts first alphabetically. Ours sorted a staging datasource to the top. Grafana remembers your last selection in your own browser, so for most of us the dashboards opened where we left them and the variable was invisible. The link pinned in our incident channel had been copied out of somebody's browser two years earlier and carried no variable in its query string, so it opened at the default. The engineer who opened it that morning had never used that dashboard before, so there was nothing remembered to correct it.
What made it convincing rather than obviously wrong is that staging runs the same code with a synthetic load generator against it. The panels were full. The shapes were plausible. There was no error state to notice, because nothing had failed. The only evidence was a dropdown in the top left corner of the page, rendered in the same grey as everything else.
The alert that fired did contain a link with the variables set correctly. Nobody clicks the alert link. People go to the dashboard they know by heart.
Environment is no longer a variable. Production dashboards live in their own folder with the datasource fixed in the JSON, and a provisioning test fails the build if anything in that folder is templated. Every production dashboard now carries a row at the top that reads an environment label out of the data itself and prints it in large type, coloured red for anything that is not production. The pinned links are generated from the repository rather than pasted from a browser.
A tool that can show you two environments needs to make which one loudest, because the failure mode is not an error message. It is a completely convincing answer to a question you did not ask.
– Sergey Shinder
