- Book: LLM Observability Pocket Guide: Picking the Right Tracing & Evals Tools for Your Team
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
You ship an agent. Traffic ramps. The traces look great in Honeycomb for the first week. Every turn, every tool call, every token sits there to scrub when something breaks. Then the bill arrives and the trace storage line is bigger than the inference line. The question that follows is the one nobody wants to answer on a Friday: what do we drop, and how do we drop it without losing the traces we actually need?
The honest answer is that 100% capture of an agent fleet stops being worth it surprisingly early. Traces are still the only thing that lets you debug a tool-call loop. The problem is the value of a trace is wildly uneven. A clean three-second turn that returned the right answer is statistical filler. A 47-second turn that ran 11 tool calls and ended in MAX_TOKENS is gold. Pay full price for both and you are subsidising the boring traces with the budget you need for the interesting ones.
This post is the math and the OTel config for getting that ratio right.
Why agent traces blow up the storage bill faster than HTTP traces
A normal HTTP request trace is maybe 10 spans. A request comes in, hits a handler, talks to a database, calls one downstream service, returns. Honeycomb, Datadog, and the rest were priced around that shape.
An agent turn is not that shape. A single user message can generate:
- 1
agent.turnparent span - 4 to 8
model.input/model.outputspans (one per loop iteration) - 5 to 30
tool.executechild spans, sometimes nested two deep when tools call other tools - A
gen_ai.usage.input_tokensattribute on every model span, often with the full prompt as a span event for replay
Add Model Context Protocol (MCP) tool boundaries and you double the count again because every MCP call is its own client + server span pair. A chatty agent doing code search across three repos lands in the 60-to-120 span range per turn. Multiply by 50,000 turns a day and you are at 3 to 6 million spans. At low-single-digit dollars per million indexed spans on the public APM pricing pages linked below, that is real money before you have routed anything anywhere.
Per Datadog's public APM pricing page (observed 2026-05), indexed spans list around the low-single-digit dollar range per million spans retained for 15 days — call it roughly $1.27/M as an illustrative figure for the math below. Honeycomb's pricing page publishes a billion-events-per-month-class quota on its paid tiers, which sounds like a lot until you do the per-turn math above and realise a mid-volume agent fleet eats it in two weeks. What matters is cost per useful trace, and at 100% sampling that ratio is dominated by the boring 90%.
Head sampling: the wrong knob first
The first instinct is head sampling: make a coin-flip decision at the start of the trace and either record it or do not. OTel's TraceIdRatioBased sampler does exactly that. The agent SDK creates the root span, the sampler decides "you are in the 10% bucket, record everything downstream; you are in the 90% bucket, drop everything."
from opentelemetry.sdk.trace.sampling import (
TraceIdRatioBased,
)
from opentelemetry.sdk.trace import TracerProvider
provider = TracerProvider(
sampler=TraceIdRatioBased(0.1),
)
Cheap, simple, deterministic. Also wrong for agents.
The defect is that head sampling decides before the trace exists. You cannot ask "did this turn fail?" or "did this turn run more than five tool calls?" at root-span creation time, because none of that has happened yet. So the 90% you drop contains the same proportion of broken turns as the 10% you keep. When a customer pings support saying "the agent looped forever yesterday at 14:32," there is a 90% chance the trace is gone. That is the trace you needed.
Head sampling is fine for a high-volume read endpoint where every request looks like every other request. Agent turns do not look like each other. Use head sampling here and you save money on storage by throwing away exactly the traces that would have paid for the bill.
Tail sampling: keep the interesting ones
Tail sampling is the opposite move. Buffer the whole trace in the OTel Collector, wait until the trace is complete, then decide based on what actually happened.
The OTel Collector ships a tail_sampling processor for this. The shape is a list of policies; if any policy says "keep," the trace is kept. The minimum useful config for an agent fleet looks like this:
processors:
tail_sampling:
decision_wait: 30s
num_traces: 100000
expected_new_traces_per_sec: 200
policies:
- name: keep-errors
type: status_code
status_code: { status_codes: [ERROR] }
- name: keep-slow
type: latency
latency: { threshold_ms: 10000 }
- name: keep-tool-heavy
type: span_count
span_count: { min_spans: 20 }
- name: keep-baseline
type: probabilistic
probabilistic: { sampling_percentage: 5 }
Four rules, in plain English: keep every trace that errored, keep every trace longer than 10 seconds, keep every trace with more than 20 spans, and keep a 5% baseline of everything else for when you need to compute "what does normal look like?"
That mix typically retains traces in the 10–20% range depending on traffic mix while keeping close to 100% of the broken turns. Your numbers will vary. The ratio of useful-trace-to-cost flips. You stop paying full freight for the boring 80%, and the on-call rotation stops opening the trace UI to find the trace they need has been thrown away.
Two things to know before you ship that config. The first is decision_wait: the collector buffers each trace in memory until it has been quiet for that long, then decides. Set it shorter than your slowest agent turn and you make the decision before all the spans arrive. Set it longer than you need and the collector's memory grows. 30 seconds is a sensible default for agent fleets where most turns finish under 15. Tail sampling also assumes every span of a trace lands at the same collector instance. That is why production deployments put a load-balancing exporter in front of a fixed pool of tail-sampling collectors. Skip the load balancer and the policy decisions get random because each collector only sees half the trace.
Reservoir sampling: the baseline that does not lie to you
The 5% probabilistic policy above is good enough for most teams. There is a sharper version when you care about statistical questions like "what is the p95 token count of a normal turn?"
The problem with naive 5% sampling is that during a traffic spike (a marketing email lands, traffic 10×s for an hour) you keep 5% of a much bigger pool, and your "baseline" sample is dominated by an unusual hour. The p95 you compute from those traces is the p95 of that hour, not of normal operation.
Reservoir sampling fixes this. Pick a fixed reservoir size (say, 10,000 traces per hour) and admit traces with probability that decreases as the reservoir fills. Every trace has equal probability of being in the final reservoir, regardless of whether it arrived during a spike or a quiet patch. The OTel tail_sampling processor does not ship a true reservoir policy. The closest practical approximation is a rate_limiting policy, which is a flow cap rather than a reservoir, but it has the property you actually want here: it bounds baseline volume regardless of input rate.
- name: keep-baseline-rate-cap
type: rate_limiting
rate_limiting: { spans_per_second: 50 }
At 80 spans per turn that is roughly 180k spans an hour, or about 2,000 baseline traces an hour, regardless of whether the upstream traffic doubled. Combined with the keep-errors and keep-slow policies above, it gives you a baseline whose volume is stable across spikes. That is what you need when you compute SLOs and want the p95 to mean something. If you need true reservoir semantics for statistical accuracy, do the reservoir admission yourself in a custom processor or upstream of the collector.
Parent-based propagation across MCP and tool boundaries
The configs above only work if the trace is connected. The instant you cross a tool-call boundary into an MCP server or a downstream microservice, you need parent-based sampling, or the child spans get their own sampling decision and the trace fragments.
OTel's ParentBased sampler is the standard wrapper:
from opentelemetry.sdk.trace.sampling import (
ALWAYS_OFF,
ALWAYS_ON,
ParentBased,
TraceIdRatioBased,
)
provider = TracerProvider(
sampler=ParentBased(
root=TraceIdRatioBased(1.0),
remote_parent_sampled=ALWAYS_ON,
remote_parent_not_sampled=ALWAYS_OFF,
),
)
Translation: at the root, capture everything (let the tail sampler in the collector decide later). For any span with a remote parent, defer to whatever the parent decided. If the parent's trace was sampled, sample me; if not, drop me. That is the only configuration where MCP tool calls show up under the agent turn that triggered them.
The mistake to avoid is independent samplers in each service. If your agent runs TraceIdRatioBased(0.1) and the MCP server it calls runs TraceIdRatioBased(0.1) independently, the joint probability of a complete trace is 1%, not 10%. After three hops you are at 0.1%. Always wrap with ParentBased once you have downstream services, and let one decision propagate.
Sample-by-tenant when one customer is the problem
The last policy is the one that pays for itself the moment a single customer files a support ticket. Sample-by-tenant: keep a higher fraction of traces from a named subset of tenants. The OTel collector supports it via the string_attribute policy:
- name: keep-debug-tenants
type: string_attribute
string_attribute:
key: tenant.id
values: [acme-corp, big-bank-inc]
enabled_regex_matching: false
Set tenant.id as a span attribute on the root span of every turn. When support pings you about acme-corp, flip them onto the debug list and you get 100% of their traces from that moment forward without changing global retention. When their issue is fixed, take them off. The tenant who is not the problem stays on the default mix.
This is also the policy that makes per-tenant SLOs honest. If your enterprise tier promises a different latency SLO than free, sample those tenants at a higher rate so the p95 you report on their dashboard is not statistical noise. A 5% baseline on a small tenant computes a p95 from maybe 50 traces a day, which is not enough to be confident in.
The cost math in one paragraph
Run all four policies (keep-errors, keep-slow, keep-tool-heavy, 5% baseline) on a typical agent fleet and you typically retain in the 10–20% range while keeping close to 100% of the traces an on-call engineer would actually open. At list price, that is illustratively in the ballpark of an 80% reduction in trace storage spend. For a fleet at 50,000 turns a day with ~80 spans per turn, you drop from 4 million ingested spans a day at 100% capture (≈120M/month) down to roughly 600k a day (≈18M/month) with the policies above. Using the illustrative $1.27 per million indexed spans figure cited earlier from Datadog's public APM pricing page (observed 2026-05), the monthly delta is roughly $150 vs $23 for the trace line, with no operational loss because the broken turns are still all there. Treat the dollar numbers as arithmetic on a public list price, not as your bill — your effective rate depends on commit, retention tier, and what your account team negotiated.
What to ship this week
If you are at 100% sampling on an agent service and the storage line on your APM bill is climbing, the order to roll out is: parent-based sampler in every service first (so traces stay connected), then a tail-sampling collector pool with the four policies above, then a tenant attribute on the root span. Skip ahead to reservoir sampling only if you compute statistics on the baseline. Most teams do not, and the rate-limiting policy is good enough.
The real win is that the traces you keep are the ones you need. Roll out the parent-based sampler tomorrow morning, then pick the policies you need before the next bill lands.
If this was useful
The LLM Observability Pocket Guide walks through the GenAI semantic conventions the configs above lean on, the collector deployment topology that keeps tail-sampling honest, and the per-tenant attribute work that lets you debug one customer without changing global retention. The agent-tracing chapter pairs directly with this post.
