When a user makes a single request to an application or clicks “checkout” on an e-commerce site, it involves multiple services, API gateway → auth service → cart service → Redis cache → inventory database → payment → notification. Traditional logs show what happened within a single service. Distributed tracing shows the entire story of a request as it traveled across all the services in the application.
The core data model has three concepts:
- Trace - journey of request. Each trace has unique trace-id
- Span - unit of work within that trace defined with a start time and duration. Could be calling another service, doing a database query. Child spans are also created
- Context - the metadata that goes with the request to tell each service that this request is part of another request. This the W3C Trace Context
The Gantt chart (known as a waterfall) immediately shows you the critical path for the application.
Auth → Cart → Inventory → Payment
Reading the Gantt Chart
Each horizontal bar represents one span of work within one service. The left edge of each bar is the start time of that work, the right edge is the finish time, and the width is the length of that span of work. The vertical axis represents the different services, and the horizontal axis is the wall-clock time between 0ms and 600ms.
The vertical lines that are indented within each service represent the parent-child relationships between those services. Each service’s work starts directly when these vertical lines start: it was called directly by its parent service.
Look at the time at when each service starts:
- Auth service starts at ~20ms and finishes at ~100ms. API Gateway must authenticate the user before continuing with any other work; Cart cannot start until Auth is complete.
- Cart service starts at ~110ms (after Auth) and makes an inline call to Redis (a child service). Cart cannot return successfully to its caller until Redis returns.
- Inventory service starts at ~270ms (after Cart) and makes an inline call to Postgres (a child service). Postgres is called inline so that the Inventory service can return to Cart with the requested inventory information.
- The Payment service starts at ~410ms (after Inventory is called) and takes 140ms to complete (the longest task after the API Gateway’s root span).
- The Notification service starts at ~555ms and is very short at ~25ms in length. This is likely a fire-and-forget service that is called asynchronously when the Purchase operation is complete.
The process strictly follows a sequence: Auth → Cart (with Redis) → Inventory (with Postgres) → Payment → Notification. None of these steps run simultaneously. That’s the main issue—the design enforces a linear chain when, in theory, some parts could operate in parallel.
Now, about why Payment is the bottleneck. A bottleneck isn’t just the slowest step; it’s the part on the critical path that, if improved, would speed up the entire workflow. Here, the longest sequence—the critical path—takes 580ms, and Payment’s 140ms sits toward the end. Cutting time in Payment directly reduces the overall response time. For example, even if you eliminated the 70ms Redis cache retrieval in Cart, Payment would still take 140ms afterward, leaving the total duration almost unchanged.
There’s also another factor: Payment involves calling an external third-party API (Stripe), which adds extra dependency and latency.
You can’t control latency here—it can jump unpredictably. Network round-trips add delays on top of Stripe’s processing time, and any outage or rate limit from Stripe stalls your whole checkout flow.
The amber highlight in the diagram shows it’s slower compared to other parts, but the real problem is where it sits—in a late spot in a chain with no parallel work and relying on an external system you can't tweak.
So, now what? Two main handles: parallelism and externalization.
First, run Inventory and Auth checks at the same time. Starting both simultaneously after the request could cut down about 200ms of sequential waiting to around 130ms, limited by whichever is slower.
Second, take Payment out of the critical path. Accept the order right away, then process payment asynchronously using a queue. This lets users see a quick “order received” response while payment happens behind the scenes—just like most large e-commerce platforms do.
Also, add a timeout and retry strategy. Since Stripe is external, its slowest 1% delays become yours too. Setting a strict timeout—say 200ms—and handling errors gracefully means a Stripe glitch won’t wreck the checkout experience.
Without this trace, you might blame the database or cache, but the waterfall view clears that up instantly.
Now, lets dive into W3C Trace Context
Before W3C settled on a standard in 2021, each had their own way of passing trace context around. Zipkin used X-B3-TraceId headers. AWS X-Ray used X-Amzn-Trace-Id. Jaeger had its own format. When requests moved between these systems, the trace often got lost. W3C Trace Context introduced two HTTP headers that all compliant systems should support
Header 1: traceparent
This is the core carrier. It encodes four fields in a single, compact header:
version — always 00 currently. Future versions can add fields.
trace-id — 128-bit identifier for the entire distributed trace. All spans across all services share this value.
parent-id — 64-bit identifier of the calling span. The receiving service uses this as its parent span when creating its own span.
flags — currently 1 byte. The lowest bit is the sampling flag (01 = sampled, 00 = not sampled).
Header 2: tracestate
Vendor extension list, an ordered comma-separated set of key=value pairs.
Each tracing vendor gets a unique key (e.g. dd for Datadog, b3 for Zipkin B3). This lets vendor-specific metadata (sampling priority, origin, session IDs) ride alongside the standard traceparent without polluting it. When a service forwards a request, it must preserve the full tracestate and may prepend its own entry at the front. Entries beyond 32 are dropped from the tail.
Headers propagation across service boundaries
The sampled flag, which shows as 01 or 00, might seem like a small detail but it actually matters a lot. Tracking every single request in a busy production system is usually too costly—for example, a service handling 100k requests per second. Sampling helps by recording just a slice of traces, yet still keeping full detail for those captured.
The sampled bit (flags 01 vs 00) plays a crucial role in tracing. Tracing every request in a high-traffic system, like one handling 100k requests per second, quickly becomes impractical. Sampling helps by recording only a subset of traces while maintaining detail for those captured.
There are three main sampling methods:
Head-based sampling decides at the root span before downstream calls. It sets the sampled flag to 01 or 00 and propagates it. This approach is straightforward and efficient but can't prioritize capturing interesting events like errors since they haven’t occurred yet.
Tail-based sampling collects all spans from every service. The collector delays the sampling decision until the full trace is received, allowing it to keep all error traces and a small fraction of normal ones. This requires a more complex, stateful collector, such as the OpenTelemetry Collector with tail sampling.
Parent-based sampling simply follows the parent’s sampling decision via the sampled flag. It’s a safe default that ensures the whole trace is either fully recorded or entirely dropped, preventing disconnected spans without a root.
A span isn’t just about its name and how long it lasts. OpenTelemetry, which leads as the open-source implementation of W3C Trace Context, sets semantic conventions—standard attribute names. This helps tracing tools interpret your data consistently, without needing extra setup.
Spans can include events, which are specific log entries tied to a moment within the span, like noting a “cache miss occurred”. They also hold links, which reference other traces—useful in async or queue situations where the producer and consumer traces are related but happen separately.
Async and Queue Propagation
Handling async and queue propagation differs from straightforward HTTP propagation. Since producer and consumer don’t operate simultaneously, you can’t rely on passing a traceparent in a request header.
Instead, the traceparent value is embedded within the message metadata—such as Kafka headers, SQS attributes, or AMQP headers. The consumer then reads this metadata and treats it as a span link, not a parent. This means the consumer’s trace connects to the producer’s but remains a separate trace-id. This approach keeps the causal link clear without letting trace information grow uncontrollably deep.
How does this fit into the OpenTelemetry stack?
As the work-horse OpenTelemetry SDK auto-instruments common frameworks like Express, Django, Spring, and net/http, automatically creating spans and managing traceparent injection and extraction. Each service uses the OTLP exporter to send span data to the OTel Collector, which processes the data—sampling, redacting sensitive info, batching, and distributing it to multiple backends at once. This vendor-neutral setup lets you switch between tools like Jaeger and Grafana Tempo without modifying your application code.
Key implementation considerations
Clock skew — spans from different machines have timestamps from different clocks. A child span with a start time before its parent is a common artifact of NTP drift. Tracing SDKs work around this by using monotonic clocks for durations and accepting some display weirdness on absolute timestamps.
Baggage — a sibling to tracestate, the W3C Baggage header (baggage: userId=alice, env=prod) lets you propagate arbitrary key-value pairs across service boundaries. It's useful for passing tenant IDs or feature flag states without touching application code, but beware: every hop re-serializes and forwards it, so large baggage values add latency.
Instrumentation gaps — if any service in the chain drops the traceparent header (an unmigrated internal proxy, a legacy service, a third-party vendor), the trace breaks. The downstream spans become "orphans" with no parent. Good tracing platforms surface these as disconnected sub-traces.
Security — traceparent is safe to expose externally. tracestate may contain internal vendor metadata you don't want leaking. Strip tracestate at your public ingress, then re-inject a fresh one for internal propagation.
The W3C standard deliberately kept the spec minimal — it defines the wire format, not how to collect, store, or visualize. That separation of concerns is what made it possible for the whole ecosystem to converge on it.
Diagrams created with the assistance of AI.
