Support told us twice that the mobile app was slow first thing in the morning. Both times I opened the dashboard for the window they gave me, found a p99 of three hundred and eighty milliseconds and an error rate of nought point two percent, and closed the ticket as not reproducible. The third time, somebody sat next to a shop manager at eight in the morning and watched it fail for eleven minutes.
Our app cancels a request after eight seconds. When a client disconnects, the connection is gone, our handler's write goes nowhere, and the middleware that records the duration never reaches the line that records it, because that line runs when a response is written. The proxy in front logs status 499, which is not a real HTTP status, and every dashboard and alert expression we own filters on a status matching two, three, four or five hundred. 499 does not match. So the slowest requests in the system removed themselves from the statistics at precisely the moment they became slow, and the more slowly a request ran the less likely it was to be counted at all.
What we were publishing as our latency distribution was a distribution of the requests that survived. Two point three percent of traffic between seven and nine in the morning was ending in a cancellation, about thirty thousand a day, and none of it existed anywhere in our numbers. Plotted, it is not subtle. It is a block of failures with clean edges, sitting on top of a report query that runs at seven.
The fix is three rules rather than a tool. Cancellations are a failure class with their own panel, their own alert and a place in the error rate, taken from the proxy rather than from us. The timing middleware records in a deferred handler with an outcome label, so an abandoned request is a measured slow request instead of nothing. And the denominator of every objective we write is requests received, never responses sent.
Anything you measure from inside your own process is a survey of the work that made it to the end. The experiences you most need to see are the ones that leave before you finish writing them down.
– Sergey Shinder
