Most failure reports in an unattended pipeline are boring. Something timed out, the agent tried again, the second attempt worked, the run closed green. Nobody looks at it. Green is the signal that permits you to stop looking.
I have started treating those runs as the most interesting artifacts the system produces, because a retry that succeeds is a confession. It says: the first attempt was not deterministic, and we do not know which of the two attempts was the anomaly.
Consider the two stories that fit the same evidence. Story one: the network hiccupped, the second call went through, the world is fine. Story two: the first call actually went through, the response was lost on the way back, and the retry created a second thing. Both produce identical logs on our side. One request, one error, one request, one success. The difference lives entirely on the other end of the wire, in a system we do not instrument.
For a while I assumed idempotency keys solved this. They solve it when the remote endpoint honors them. Half the surfaces an agent touches do not — publishing endpoints, notification hooks, anything that accepts a payload and returns a bare status. So the guarantee we thought we bought was a guarantee about a well-behaved subset, and the subset was never enumerated. It existed only as an assumption inside the retry wrapper.
The second thing a successful retry hides is the shape of the failure. When an agent retries a step three times and the third works, we record it as one success. What actually happened is that the system spent two-thirds of its attempts producing nothing, and that ratio is a health metric we deleted at the moment we chose to report the outcome instead of the attempts. A pipeline where every step succeeds on the third try and a pipeline where every step succeeds immediately look the same on a dashboard that counts completions. They are not the same pipeline. One of them is about to stop working and has been telling you for weeks.
So we started counting attempts, not outcomes. The number that matters is not "did the run finish" but "how much work did it take to make the run finish." A run that finished after eleven internal retries is a failing run that happened to end well. Once that number was visible, it moved. It moved before anything broke. It moved when an upstream provider quietly tightened a rate limit, days before we hit a hard wall, because our attempt count crept from 1.02 to 1.4 while our success rate stayed at 100 percent.
There is a subtler cost. Retries teach the agent the wrong lesson about its own reliability. If the wrapper absorbs transient failures silently, the agent's model of the world is that its tools always work. It never develops the habit of verifying. It publishes, gets a success, and moves on — and when a retry has produced a duplicate, the agent is structurally incapable of noticing, because the duplicate was created by infrastructure operating below the level at which the agent reasons. The mistake is invisible to the only component that could correct it.
What I want from a retry layer now is not silence but a receipt. Every absorbed failure should surface as a small annotation attached to the run: this step required more than one attempt, here is the error we swallowed, here is the number of times. Not an alert. Alerts on retries would be unbearable noise. A receipt — something a weekly review can aggregate, something that turns invisible degradation into a slope.
The uncomfortable version of this: a system with aggressive retries and no attempt telemetry is not more reliable than one without. It is equally unreliable and better at hiding it. You have converted a category of visible failure into a category of silent duplication and slow decay, and you have paid for that conversion with the ability to see it coming.
Green runs are not evidence of health. They are evidence that the reporting threshold was not crossed. Those are different claims, and unattended systems will only ever hand you the second one.
