When Better Latency Metrics Hide Worse Reliability

A system completes 93% of interactive requests with a p95 latency of 10 seconds.

Another completes 97% with a p95 latency of 12 seconds.

Which one performed better?

The obvious answer seems to be the first system. Twelve seconds is worse than ten.

But there is a problem hidden in that comparison:

The latency percentile is calculated only over requests that succeeded.

The second system completed requests that the first system did not.

Once systems admit, reject, retry, or shed different subsets of traffic, successful-request latency stops being an apples-to-apples comparison by itself.

This matters particularly under overload.

And it matters a lot for LLM infrastructure.

The denominator changed

Latency percentiles are extremely useful.

Google's SRE guidance recommends percentiles precisely because averages can obscure the long tail of request latency. A system with a reasonable average can still have a terrible experience for its slowest requests.

But every percentile has a population.

Consider 100 requests sent to two systems.

System A completes 90:

Requests offered:     100
Requests completed:    90
Requests failed:       10
Successful p95:       4.0s

System B completes 99:

Requests offered:     100
Requests completed:    99
Requests failed:        1
Successful p95:       5.0s

Looking only at latency produces a simple conclusion:

System A is faster.

Looking only at successful completion produces another:

System B is more reliable.

Neither metric is wrong.

They measure different properties.

The mistake is turning either one into a complete description of the system.

Overload makes this especially dangerous

Under ordinary load, two systems may successfully process almost exactly the same requests.

Their latency distributions are therefore reasonably comparable.

Under overload, admission policy starts changing the population.

A system can:

• reject requests immediately;
• queue them;
• retry them;
• prioritize one workload class;
• shed expensive work;
• reserve capacity for selected traffic;
• allow requests through until an upstream dependency rejects them.

Those decisions determine which requests eventually appear in the successful-request latency histogram.

That means the latency measurement becomes conditional:

Latency, given that the request succeeded.

That is not the same metric as:

What happened to a request offered to the system?

The distinction becomes important as failure rates diverge.

A real example

I ran into this while evaluating admission-control strategies for mixed LLM workloads.

The benchmark compared four approaches under the same deterministic workload:

1. No application-level control
2. A static cap
3. Redis-coordinated admission
4. Local admission backed by distributed capacity coordination

The workload intentionally exceeded the provider's 32-request capacity envelope and mixed latency-sensitive interactive requests with background batch work.

At the zero-coordinator-latency rung, measured across eight matched seeds, two results initially seemed contradictory.

Redis produced better successful-request latency:

If I stopped there, Redis would clearly look better.

But the completion results looked different:

The local strategy completed substantially more interactive work while performing fewer retries and shedding fewer attempts.

Redis completed fewer interactive logical requests, but the requests that did complete reached their first token faster.

Both observations are true.

So which system actually won?

That question is too broad.

Redis won one dimension of the experiment:

responsiveness among successfully completed requests.

The local admission strategy won another:

successful completion of protected interactive work.

It is tempting to collapse those into a single ranking.

Doing so throws away useful information.

If your primary objective is minimizing latency for requests that survive overload, the Redis result may be preferable.

If your primary objective is preserving interactive completion when capacity becomes scarce, the other policy may be preferable.

And if batch throughput matters most, the answer changes again: Redis also completed considerably more batch work in this particular benchmark.

The benchmark therefore does not establish a universal winner.

It exposes a tradeoff.

That is exactly what a useful benchmark should do.

The survivor effect

There is a deeper measurement issue here.

Imagine a workload with ten requests.

Their hypothetical service times, if all could run successfully, might be:

1s
1s
1s
2s
2s
2s
3s
5s
8s
15s

Now imagine System A admits all ten.

System B admits the first seven and rejects the last three.

System B's successful-request latency distribution looks excellent.

But it did not make the 5-, 8-, and 15-second requests faster.

It removed them from the population being measured.

This is an intentionally simplified example, but it demonstrates the problem.

A successful-request latency percentile tells you about the survivors.

It does not tell you what happened to the requests missing from the sample.

This resembles a familiar statistical problem: conditioning on an outcome can change the composition of the population you are comparing.

In infrastructure benchmarks, admission and load shedding can create exactly that situation.

Be careful with the stronger interpretation

There is a tempting explanation for the benchmark above:

The local strategy had worse p95 latency because it successfully carried harder requests that Redis rejected.

That is plausible.

It is not fully established by the aggregate numbers.

The benchmark did include requests with substantially different sizes—interactive request sizes varied by roughly 25× to 32×—and token pressure was a real admission constraint. The two systems also made different rejection decisions.

But proving the stronger statement would require matched request-level counterfactual evidence.

For each request rejected by one policy and completed by another, we would want to know what its latency would have been had the rejecting system admitted it.

We cannot observe both outcomes in the same run.

So the defensible conclusion is narrower:

When completion rates differ, successful-request latency percentiles describe different populations and therefore should not be interpreted independently of completion.

That claim does not require speculation.

The measurements themselves establish it.

This is why availability and latency belong together

SRE practice already points in this direction.

Google treats both availability and latency as distinct service-level indicators. Availability measures the proportion of requests that succeed; latency measures how quickly requests are served.

Neither replaces the other.

Suppose a release changes your dashboard from:

Success: 99.5%
p95:     800ms

to:

Success: 94.0%
p95:     500ms

Celebrating the 300ms latency improvement would be premature.

Maybe the service became dramatically more efficient.

Or maybe it stopped serving a meaningful portion of its workload.

You have to investigate both.

The same principle applies when comparing architectures.

Retries distort the picture further

Retries introduce another layer.

Suppose a logical user request is rejected twice and succeeds on its third attempt.

At the attempt level, you may observe:

Attempt 1: rejected
Attempt 2: rejected
Attempt 3: TTFT 900ms

If the dashboard reports only the final successful provider attempt, the user appears to have received a response with a 900ms time to first token.

They didn't.

They experienced rejection, backoff, another rejection, more backoff, and then 900ms to first token.

A benchmark needs to distinguish at least three concepts:

• provider-attempt latency;
• successful logical-request latency;
• end-to-end user-observed latency.

Otherwise retry-heavy systems can appear healthier than they are.

This matters operationally as well as statistically. Google warns that retries can amplify overload and contribute to cascading failure, which is why exponential backoff and jitter are standard overload protections.

In the LLM benchmark, this was one reason I tracked retry amplification alongside latency.

Redis produced lower successful-request TTFT, but required 1.478 attempts per logical interactive request at the median.

The local strategy recorded 1.339.

Neither number means much without the other metrics around it.

Rejections need context too

Even rejection count can mislead.

One admission controller may reject an attempt locally before expensive work begins.

Another may admit that attempt, consume provider capacity, and eventually receive a provider-side 429.

A third may queue the request until the caller times out.

All three can result in an unsuccessful request.

Operationally, however, they are very different outcomes.

Early rejection is often deliberate overload protection.

Google's SRE guidance recommends graceful load shedding when a service exceeds sustainable capacity rather than allowing overload to turn into extreme latency or cascading failure.

So "fewer rejects" is not automatically better either.

What ultimately matters is what happens to the logical workload:

Was the important request eventually completed?

How long did the caller wait?

How many attempts did it require?

How much capacity was consumed along the way?

What other work was displaced?

Use a scorecard, not a hero metric

For overload experiments, I now prefer to think in terms of a scorecard.

For latency-sensitive traffic, I want at least:

Offered logical requests
        |
        +-- completed
        |
        +-- exhausted / failed
        |
        +-- attempts
              |
              +-- admitted
              +-- locally rejected
              +-- upstream rejected

And then I measure:

Completion

• logical-request success rate;
• successful goodput;
• exhaustion rate.

User experience

• end-to-end p50;
• end-to-end p95;
• TTFT p50;
• TTFT p95.

Admission behavior

• local rejects;
• queue wait;
• token-budget rejects;
• concurrency-bound rejects.

Amplification

• attempts per logical request;
• retries per request;
• upstream 429s.

Capacity

• peak occupancy;
• in-flight concurrency;
• in-flight token exposure.

And for mixed workloads, every important result should be segmented by workload class.

An aggregate 95% success rate is not reassuring if interactive success is 80% and low-priority batch success is nearly 100%.

Always print the denominator beside the percentile

A simple reporting habit prevents a surprising amount of confusion.

Instead of:

TTFT p95: 2.9s

report:

Interactive success: 92.7%
Successful-request TTFT p95: 2.9s

Even better:

Offered:                   270 logical requests
Completed:                 250
Interactive success:       92.7%
Successful-request p95:    10.3s
Successful-request TTFT:    2.9s
Attempts/logical request:   1.478

Now the reader can see the population behind the latency statistic.

The word successful-request is worth keeping.

It is a reminder that failures are absent from the percentile.

Matched workloads matter

The measurement problem becomes worse when benchmark workloads themselves differ.

If one system receives smaller requests, fewer long prompts, or a different arrival sequence, latency comparisons tell you very little.

This is why controlled overload experiments should use matched workloads.

In the benchmark described above, each strategy was run against matched seeds, and results were reported across eight seeds rather than selecting one representative-looking execution. The broader coordinator-distance experiment ran eight matched seeds at four distance rungs, producing 32 recorded proof combinations.

Repeating runs matters because overload is timing-sensitive.

Queue ordering changes.

Retries shift arrival patterns.

Capacity becomes available at slightly different moments.

A single run can produce a compelling graph that turns out not to represent the system reliably.

Don't optimize the dashboard

There is a dangerous feedback loop in performance engineering:

1. Choose an important metric.
2. Optimize the system.
3. Watch the metric improve.
4. Assume the system improved.

Usually that works.

Sometimes the optimization changes what gets counted.

Admission control is a particularly clear example.

A stricter policy can improve the latency distribution simply by excluding more work.

A more permissive policy can make latency look worse while completing requests that would otherwise disappear from the successful population.

Neither outcome is inherently desirable.

The business requirement determines which tradeoff is acceptable.

If a user clicks "Generate" in an interactive application, I might willingly accept an additional second of tail latency to materially improve the probability that the request completes.

For a latency-sensitive trading system, that trade might be unacceptable.

For an asynchronous document-processing pipeline, throughput might dominate both.

Metrics don't decide this.

They tell us what trade we made.

A better question

When comparing overloaded systems, don't ask only:

Which one has the lower p95?

Ask:

Lower p95 over which requests?

Then ask:

What happened to the rest?

That small change in framing makes overload benchmarks much harder to misread.

Latency percentiles remain indispensable. Tail latency matters, and averages absolutely can hide serious performance problems.

But latency is only one part of the outcome.

Once two systems make different admission, retry, or load-shedding decisions, their successful-request distributions may no longer contain equivalent populations.

At that point, a prettier latency number can coexist with a worse reliability result.

The percentile isn't lying.

We're just asking it to tell us something it doesn't measure.