Originally published on Loop & Retry — field notes on building LLM agents that survive production.
The demo felt instant. The agent answered in about four seconds, every time you ran it on stage. Then you shipped it, and the support queue filled with "it hangs." Nothing was broken. Your average latency really was four seconds. The problem is that nobody experiences the average — they experience one run, and one run is a roll of the dice across every step. This post is about why the latency you ship is the tail, not the mean, and why adding steps makes the tail worse in a way that feels unfair until you see the arithmetic.
This is the latency half of a pillar whose other half I already wrote about in retry budgets. Cost compounds multiplicatively across steps; latency compounds too, but through a different mechanism, and the fix is different.
The mean is the number nobody feels
Here's the intuition to kill. Your agent takes N steps. Each step is a model call plus a tool call, and each takes some time that varies run to run — usually fast, occasionally slow, because model latency has a long right tail (a slow token, a cold route, a retried request underneath you). You measure the average step at, say, 500ms, multiply by 8 steps, and report "4 seconds."
That number is real and it is useless. The user doesn't run your agent a thousand times and average the wall clock. They run it once. And a single run is the sum of eight independent draws from a right-skewed distribution — which means the run is slow whenever any one of its eight steps happens to land in the tail. With eight steps, the chance that at least one lands in its slow 10% isn't 10%. It's 1 − 0.9⁸ ≈ 57%. More than half your runs contain a step that was individually slow, and that step sets the pace of the whole run.
Let's measure it instead of hand-waving.
A latency model you can run
This is deliberately small. It draws a per-step latency from a lognormal (the standard shape for "usually fast, sometimes much slower"), sums the steps into a run, and reports what the mean hides.
import random, statistics
N = 8 # steps to finish the task
MU = 6.0 # lognormal mu -> median step ~ exp(6.0) = 403ms
SIGMA = 0.6 # tail heaviness; bigger = fatter slow tail
def step_ms():
return random.lognormvariate(MU, SIGMA)
def run_ms():
return sum(step_ms() for _ in range(N))
def pct(xs, q):
return sorted(xs)[int(q * len(xs)) - 1]
runs = [run_ms() for _ in range(200_000)]
steps = [step_ms() for _ in range(200_000)]
print(f"step mean={statistics.mean(steps):6.0f}ms p50={pct(steps,.50):6.0f} "
f"p95={pct(steps,.95):6.0f} p99={pct(steps,.99):6.0f}")
print(f"run mean={statistics.mean(runs):6.0f}ms p50={pct(runs,.50):6.0f} "
f"p95={pct(runs,.95):6.0f} p99={pct(runs,.99):6.0f}")
Running it:
step mean= 485ms p50= 405 p95= 1088 p99= 1646
run mean= 3862ms p50= 3755 p95= 5493 p99= 6481
Look at what happened to the ratios. A single step's p99 is 4× its median (1646 vs 405) — that's the fat tail you expected. But the run's p99 is only 1.7× its median (6481 vs 3755). The tail got relatively tamer at the run level, because summing eight independent draws averages out: it's unlikely all eight are slow at once, so the extremes partly cancel.
That sounds like good news, and it's the first thing people get wrong. The relative tail shrinks, but the absolute gap between "typical" and "slow" grows. Your median user waits 3.8s; your p99 user waits 6.5s — nearly three seconds longer than the number you demoed. The mean (3.9s) sits just above the median and describes no one's actual experience of the slow path. You cannot budget a timeout, a loading spinner, or an SLA off the mean. You have to budget off the p99, and the p99 is a different animal.
The tail you can't average away
The summing-averages-out effect has a hard limit: it only works when steps are independent and none of them dominates. Two things break that, and both are common in agents.
One step with a heavier tail poisons the whole run. Suppose seven of your steps are quick model calls but one is a tool that hits a flaky downstream API with a genuinely fat tail. Bump just that step's sigma:
def run_ms_one_bad():
total = 0.0
for i in range(N):
sigma = 1.3 if i == 3 else SIGMA # step 3 is the flaky tool
total += random.lognormvariate(MU, sigma)
return total
run (uniform tails) p50=3755 p95=5493 p99= 6481
run (one fat step) p50=3916 p95=7170 p99=11778
The median barely moved. The p99 jumped more than five seconds. One brittle step, and averaging no longer saves you — that step is the tail now. This is the latency mirror of a lesson from the cost side: failure isn't uniform, and neither is slowness. Find the one worst step before you optimize the average of all of them.
Retries live inside these numbers. Every table above assumed each step runs once. A step that fails and retries doesn't just cost tokens — it serializes another full round-trip onto the critical path, and the retry is correlated with slowness (timeouts are a common failure, and a timeout is by definition a slow step that then runs again). Retries don't add to the tail; they are the tail. If you tuned your retry policy purely on cost, you set your latency p99 without looking at it.
The two levers that actually move it
The model is a toy, but the levers it exposes are real and ordered by leverage.
Take steps off the critical path. The single biggest lever is turning a sum into a max. If two steps don't depend on each other — two retrievals, a lookup plus a validation, three independent tool calls — running them concurrently changes the run's latency from
a + btomax(a, b). Crucially,maxof two tail draws is far better than their sum: you wait for the slower of two, not the total of both. Most agent loops are needlessly serial because the framework's default is "one tool call per turn." Auditing for parallelizable steps is the highest-return latency work you can do, and it costs you nothing at the token level.Fix the worst step, not the average step. As the fat-step table showed, one heavy-tailed dependency sets your p99 single-handedly. A timeout-and-fallback on that step (return a degraded-but-fast result instead of waiting out the tail) buys more than shaving 50ms off every other step combined. You cannot know which step it is without per-step latency instrumentation — so measure per step, at the p95/p99, not just the run total. (Trajectory-level measurement is its own discipline.)
Two levers I'd reach for only after those: stream so that perceived latency (time to first token) decouples from total latency — a user watching output appear tolerates a slow tail far better than one staring at a spinner; and cap the trajectory length, because every step you add is another independent chance to draw from the tail, and the arithmetic on that only goes one way.
The number that matters isn't your average latency. It's your p99, it's set by your slowest step and your most serial dependency, and both of those are things you chose. Measure the tail before you promise anyone the mean.
The model here is a back-of-envelope Monte Carlo, not a benchmark of any specific system — the lognormal shape and the step count are stated so you can swap in latencies you actually measured. The lesson (runs are sums, the tail is what ships, parallelism turns sum into max) is provider-independent; the specific millisecond figures are illustrative.
