Why Your p99 Latency Looks Fine but Users Still Complain

Your dashboards are green. p99 is under 300ms. Error rate is below 0.1%. And yet — Slack is blowing up with user complaints about slowness.

Sound familiar?

The problem isn't your monitoring. The problem is what your monitoring can't see. Metrics are summaries. And summaries, by definition, lose information.

Let's break down exactly why this happens — and what you need to see the full picture.

The p99 Illusion

When we say "p99 latency is 280ms", we mean: 99% of requests completed in 280ms or less. That sounds great. But consider what that actually hides:

At 100 requests/second, p99 = 1 user per second getting a slow response. That's 3,600 people every hour. At 1,000 req/s it's 36,000 per hour. These users aren't in your "fine" zone — they're the ones opening support tickets.

But here's what makes it worse: p99 is computed across all requests, all endpoints, all user types, all geographies. That aggregate buries signal under noise.

Your /api/checkout might have a p99 of 2.1 seconds while your /api/health pulls the overall number down to 280ms. You'll never know by looking at a single dashboard metric.

The math of fan-out: In a microservices architecture where a single user request fans out to 5–10 downstream services, the probability of at least one of them hitting a slow response compounds fast. If each service has a 1% chance of responding slowly, a request touching 10 services has roughly a 10% chance of being slow overall. Your users feel this. Your p99 dashboard doesn't.

What Aggregated Metrics Can't Tell You

Metrics — whether from Prometheus, StatsD, or any other system — are fundamentally aggregated. By the time a number lands on your dashboard, it's already lost:

• Which specific user had a slow request
• Which code path was executed (feature flag A or B? logged-in or anonymous?)
• Which downstream service introduced the latency
• What was happening in that process at that moment (GC pause? connection pool exhausted? cold cache?)
• How the latency cascaded across service boundaries You can add more metrics to try to cover more ground — but you'll end up in dashboard hell, staring at 40 graphs and still not knowing why something is slow.

The Three Patterns That Fool p99

1. Bimodal distributions

Your response times may cluster into two distinct populations: fast requests (cached data, warm connections) and slow ones (cold cache, serialized writes, DB locks). The p99 number is an artifact of the ratio between those two populations. If 2% of requests are hitting the slow path, p99 looks fine. But those users have an awful experience.

2. Segment-specific degradation

Certain users systematically hit slower paths:

• Power users with large data sets
• Users in specific geographic regions routing through a slow VPN or edge node
• Users on a specific account plan using a different database shard
• Mobile users over 3G with larger payload sizes If you're not slicing metrics by user segment, you're averaging them out of existence.

3. Periodic spikes that p99 misses

Tail latency spikes caused by JVM garbage collection, database autovacuum, or scheduled jobs may last 200–500ms but fire every 60 seconds. Depending on how your metrics are scraped and averaged, these blips might not register in p99 at all — but every user who happens to fire a request during that window gets hit.

What Distributed Tracing Reveals

Distributed tracing doesn't aggregate. It records the individual execution path of each request as it flows through your system — every function call, every database query, every external HTTP call, each one timestamped and linked together into a single trace.

A trace gives you:

• Exact duration of every span — you can see that db.query took 1.8 seconds on this specific request
• Service-level breakdown — which microservice introduced latency
• Causal chain — span A called span B which called span C; the root cause is visible
• Request-level attributes — user ID, feature flag, region, plan tier This is the difference between "checkout is slow sometimes" and "checkout is slow for users with more than 50 items in cart when hitting the EU database replica during business hours."

A Real Scenario

Imagine you run an e-commerce platform. Users report the cart page is slow, but your p99 on /api/cart is 180ms. Everything looks fine.

With tracing, you pull up slow traces (anything over 500ms on that endpoint). You filter by duration. You find 800 traces in the last hour that exceeded 1 second.

You open one. The waterfall view shows:

GET /api/cart                        [1,240ms total]
  ├── auth middleware                [12ms]
  ├── fetch user session             [8ms]
  ├── fetch cart items               [14ms]
  └── fetch product details (x47)   [1,190ms]  ← 47 sequential DB calls

There it is. Users with large carts trigger an N+1 query. 99% of users have small carts, so p99 doesn't catch it. But the 1% with 40+ items — exactly your most engaged customers — are waiting over a second on every cart load.

No amount of metric dashboards would have found this. One trace did.

Setting This Up with OpenTelemetry

OpenTelemetry is the CNCF standard for instrumenting your applications. It's vendor-neutral, widely supported, and means you're not locked in to any single backend.

For Node.js, auto-instrumentation takes about 5 minutes:

npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http

// tracing.js — load before anything else
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'https://otlp.uptrace.dev/v1/traces',
    headers: {
      'uptrace-dsn': process.env.UPTRACE_DSN,
    },
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

That's it. Express routes, HTTP clients, PostgreSQL, Redis, gRPC — all instrumented automatically. Every incoming request becomes a trace with full span detail.

For Python, Go, Java, and others — the OpenTelemetry docs have language-specific getting started guides. Uptrace also has detailed quickstart guides for Go, Python, Ruby, and more.

Finding the Slow Traces in Uptrace

Once traces are flowing, Uptrace gives you several ways to find the ones that matter:

Filter by duration — sort spans by slowest first. No query language needed to find your worst offenders.

Use UQL to drill in — Uptrace Query Language lets you write queries like:

where span.duration > 1s
  and span.status_code = "ok"
  and user.plan = "enterprise"
group by span.name

This surfaces slow-but-successful requests for enterprise users — exactly the segment most likely to churn if performance degrades.

Service graph — the service dependency graph shows you which service-to-service calls have the highest latency, so you can immediately see which hop in your architecture is the bottleneck.

Correlate with metrics and logs — a single slow trace can be opened alongside its logs and the infrastructure metrics from that moment. Is the database CPU spiking at the same time? Was there a deployment 20 minutes earlier? Uptrace surfaces this context in one place.

The Right Mental Model

Think of metrics, logs, and traces as three different lenses:

p99 latency is a metric. It's great for trend lines and alerting thresholds. But it's the wrong tool for diagnosing why users are experiencing latency. For that, you need traces — individual, end-to-end records of what actually happened.

The OpenTelemetry observability primer calls this the "three pillars of observability." In practice, it means you need all three, but traces are the pillar most teams underinvest in — and the one that pays off most when users complain.

Practical Next Steps

1. Instrument your critical paths with OpenTelemetry auto-instrumentation. You don't need to manually add spans to get value — auto-instrumentation covers HTTP, DB, and messaging by default.
2. Pick a trace backend that lets you query and filter individual traces, not just aggregates. If you want to self-host, Uptrace is open-source and deploys on a single Docker container.
3. Set up a slow-trace alert — any trace over 1s (or your SLO threshold) should surface automatically, not require a user to report it.
4. Add key attributes to your spans — user ID, plan tier, feature flags. The more dimensions you attach, the faster you can segment slow traces by affected cohort.

5. Build a latency SLO using trace data, not just metrics. Measure the percentage of individual requests that complete within your target, bucketed by endpoint and user segment.

Your p99 is a lagging indicator that averages away the problems your most valuable users are experiencing. Distributed tracing turns those invisible complaints into a precise, reproducible, debuggable stack of evidence.

The question isn't whether users are experiencing slow requests. They are. The question is whether you can see which ones.

Uptrace is an open-source APM built on OpenTelemetry. It stores traces, metrics, and logs and lets you query them together — so when a user complains, you can find the exact trace in seconds. Get started free →