OpenTelemetry Sampling: Head-Based and Tail-Based, Explained

Sampling records a subset of traces instead of all of them. A service handling 50,000 requests per minute produces more trace data than it is useful to store, and most of those traces look identical. Sampling keeps the ones worth keeping.

The decision has two dimensions: where it is made — in your application or in the Collector — and when — before a trace starts or after it finishes.

Quick start

Most applications should start here: sample a fixed percentage, decided at the root of the trace, with children following the root's decision. No code changes required.

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1   # keep 10% of traces

The SDK default is parentbased_always_on — every trace is kept. parentbased_traceidratio is the first thing to change in production, and two properties make it the safe choice:

• The decision is derived from the trace ID, so every service reaches the same conclusion about the same trace without coordinating. You get whole traces, not fragments.
• Children respect the parent's decision, so a sampled trace stays sampled all the way down.

Raise the ratio until the data volume is uncomfortable, then look at the strategies below.

How a sampling decision is made

1. Request arrives.
2. Head-based decision in the SDK. Dropped → no spans are ever created.
3. Spans exported (if sampled).
4. Tail-based decision in the Collector. Dropped → the whole trace is discarded.
5. Backend — kept traces land here.

Head-based decides when the root span starts. It is cheap — dropped traces cost nothing, because their spans are never created — but it decides before knowing anything about the outcome. It cannot preferentially keep errors or slow requests, because at that moment neither has happened.

Tail-based decides after the whole trace has been collected. It can keep exactly the interesting traces, but every span must be produced, exported, and buffered until the decision, which costs CPU, network, and Collector memory.

They compose. A common production setup is head-based sampling to cut obvious volume, then tail-based to select from what remains.

Head-based sampling

Built-in samplers

The parentbased_ variants matter more than they look. Without them, each service samples independently, and a trace crossing five services is kept by some and dropped by others — producing traces with holes in them.

Configuration

Environment variables cover the standard samplers:

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.25

In code, when you need something the environment variables cannot express:

Go

import "go.opentelemetry.io/otel/sdk/trace"

provider := trace.NewTracerProvider(
    trace.WithSampler(
        trace.ParentBased(trace.TraceIDRatioBased(0.25)),
    ),
)

Python

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

provider = TracerProvider(
    sampler=ParentBased(root=TraceIdRatioBased(0.25))
)

Node.js

const { NodeSDK } = require('@opentelemetry/sdk-node');
const {
  ParentBasedSampler,
  TraceIdRatioBasedSampler,
} = require('@opentelemetry/sdk-trace-base');

const sdk = new NodeSDK({
  sampler: new ParentBasedSampler({
    root: new TraceIdRatioBasedSampler(0.25),
  }),
});

Custom samplers

When the rate should depend on what is being traced — health checks at 1%, checkout at 100% — implement the sampler interface. Two mistakes are common enough to call out: forgetting that the interface requires a description method, and constructing a new delegate sampler on every span instead of once at startup.

import (
    "strings"

    "go.opentelemetry.io/otel/sdk/trace"
)

type RouteSampler struct {
    noise    trace.Sampler // health checks, static assets
    standard trace.Sampler // everything else
}

func NewRouteSampler() *RouteSampler {
    return &RouteSampler{
        noise:    trace.TraceIDRatioBased(0.01),
        standard: trace.TraceIDRatioBased(0.1),
    }
}

func (s *RouteSampler) ShouldSample(p trace.SamplingParameters) trace.SamplingResult {
    name := p.Name

    // Always keep payment flows.
    if strings.Contains(name, "payment") || strings.Contains(name, "checkout") {
        return trace.AlwaysSample().ShouldSample(p)
    }

    if strings.Contains(name, "health") || strings.Contains(name, "static") {
        return s.noise.ShouldSample(p)
    }

    return s.standard.ShouldSample(p)
}

func (s *RouteSampler) Description() string { return "RouteSampler" }

Wrap it in trace.ParentBased(NewRouteSampler()) so downstream services still follow the root decision. A custom sampler that ignores the parent produces fragmented traces.

Matching on span name is a blunt instrument — names are set by instrumentation libraries and change between versions. Where the routing decision is available as an attribute, match on that instead.

Tail-based sampling

Head-based sampling cannot keep failed or unusually slow traces, because neither fact exists when the decision is made. Tail-based sampling waits until the spans of a trace have arrived, then decides.

This runs in the Collector, using the tail sampling processor.

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 1000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      - name: slow
        type: latency
        latency:
          threshold_ms: 2000

      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

service:
  pipelines:
    traces:
      processors: [tail_sampling]

Policies are evaluated as OR: a trace is kept if any policy says keep. The configuration above keeps every error, every trace slower than two seconds, and 10% of everything else.

The constraint that breaks tail sampling

Every span of a trace must reach the same Collector instance. The processor buffers spans by trace ID in memory; a span that lands on a different replica is invisible to the decision. With more than one Collector replica behind an ordinary load balancer, traces are split, sampling decisions are made on partial data, and the traces you keep have holes.

This is the most common way tail sampling fails in production, and it fails quietly — the pipeline reports no errors.

The fix is a two-layer Collector deployment. The first layer receives traffic and routes by trace ID; the second layer runs the tail sampling processor:

exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: otel-collector-sampling-headless.observability.svc.cluster.local

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [loadbalancing]

The second layer must be addressed by a headless service, so the exporter resolves individual pod addresses rather than a single virtual IP.

Sizing

• decision_wait must exceed the duration of your slowest traces. Set it to 10s when p99 latency is 3s and spans arrive late, and a trace still in flight when the timer expires is judged on the spans that arrived.
• num_traces bounds memory. The processor holds this many traces in memory; multiply by your average spans per trace and span size to estimate the footprint.
• expected_new_traces_per_sec presizes internal buffers. Setting it far below reality causes reallocation under load.

Rate limiting

Rate limiting caps throughput rather than sampling a proportion: at most N traces per second, regardless of incoming volume. It protects a backend from traffic spikes, which proportional sampling does not — 10% of a tenfold spike is still a tenfold increase.

It is a poor primary strategy. Which traces survive depends on arrival order rather than on anything meaningful, and the effective sampling rate varies with load, which makes adjusted counts inaccurate. Use it as a ceiling alongside ratio-based sampling, not instead of it.

Most backends, including Uptrace, apply rate limiting automatically when necessary.

Sampling probability and adjusted counts

Sampling changes what your data means. If you keep 10% of traces and count 50 errors, the system produced roughly 500. Recovering the original figure requires knowing the sampling rate — the adjusted count, the number of real events each recorded event represents.

This only works if the rate is known to whoever does the counting. A trace sampled at 10% in one service and 50% in another has no single rate, which is what consistent probability sampling addresses: the sampling threshold travels in the tracestate header, so any participant can recover the effective probability.

The mechanism encodes a rejection threshold in an ot=th: entry in tracestate, with randomness taken either from an explicit rv sub-key or from the trailing bytes of the trace ID.

⚠️ Consistent probability sampling is still in development in the specification. Until SDKs implement the W3C Trace Context Level 2 randomness requirements, the guidance is to keep using parent-based sampling with TraceIdRatioBased at the root — mixing the two during the transition produces incomplete traces.

Choosing a strategy

Two rules survive most designs. Sample at the root and let children follow, or accept fragmented traces. And keep the number of rules small — every rule is a reason for a trace to be missing when you go looking for it.

A worked configuration

A service handling 15,000 requests per minute, where checkout must always be traced, errors must never be lost, and browsing traffic is noise.

In the application — cut obvious volume before it is produced:

export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.2

Combined with the RouteSampler above if checkout needs a guarantee at the source.

In the Collector — select from what survives:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]

      - name: slow
        type: latency
        latency:
          threshold_ms: 1000

      - name: checkout
        type: string_attribute
        string_attribute:
          key: http.route
          values: [/checkout, /api/orders]

      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

The effective rate for ordinary browsing traffic is 20% × 5% = 1%, while errors and checkout requests that survive the head-based stage are kept in full. Note the ordering effect: head-based sampling runs first, so a checkout request dropped at the root never reaches the tail policy that would have kept it. This is why flows that must never be lost need a rule at both layers.

Sampler APIs by language

The environment variables work everywhere. Reach for the API when the rate has to depend on something the variables cannot see.

To implement a custom sampler, satisfy the SDK's sampler interface: a method that returns the decision, and a method that returns a human-readable description of the sampler. Forgetting the second one is the usual reason a custom sampler fails to compile in Go.

What's next?

This piece focuses on the sampling decision itself. For the surrounding pieces:

• OpenTelemetry distributed tracing — what is being sampled
• OpenTelemetry Collector — where tail sampling runs
• OpenTelemetry context propagation — how the decision travels between services
• OpenTelemetry APM — why sampling is the main lever on cost

This post originally appeared on the Uptrace blog as part of our OpenTelemetry guide. Uptrace is an open-source, OTel-native APM built on ClickHouse.