A few years into running a JVM backend in production, I stopped trusting dashboards just because they were green. The numbers weren't wrong, exactly — CPU fine, heap fine, request count nominal — but "green dashboard, angry customer" kept happening often enough that I started keeping a private list of incidents where every chart we had swore everything was fine. It was always the same story: we were watching the machine, not the request. A service can sit comfortably under every resource limit it has and still be failing the one thing anyone actually cares about, which is whether this specific customer's call came back correctly, and on time.
That gap between "the infrastructure looks healthy" and "the system is doing its job" is where most observability budget quietly goes to waste. And closing it doesn't take a new tool. It takes logging the right three events per request, and pointing a dashboard at them the right way.
What actually pays off
Structured logging with a correlation ID that survives the whole request. I don't mean "we use JSON logs" — plenty of teams do that and still can't answer "show me every log line for this one request" without grepping across five services and guessing at a timestamp window. What actually works is assigning a correlation ID right at the edge, dropping it into the logging context (MDC, if you're on the JVM) before anything else runs, and carrying it through every outbound call — HTTP header, message header, whatever the transport happens to be. Three services downstream, it's still there.
@Component
public class CorrelationIdFilter extends OncePerRequestFilter {
private static final String HEADER = "X-Correlation-Id";
@Override
protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res,
FilterChain chain) throws IOException, ServletException {
String correlationId = Optional.ofNullable(req.getHeader(HEADER))
.orElseGet(() -> UUID.randomUUID().toString());
MDC.put("correlationId", correlationId);
res.setHeader(HEADER, correlationId);
try {
chain.doFilter(req, res);
} finally {
MDC.clear(); // thread pools reuse threads; an uncleared MDC leaks into the next request
}
}
}
That finally block is the part people miss, and it's the part that actually matters. Application servers reuse threads across requests, and MDC is thread-local. Forget the clear and, sooner or later, you'll get log lines tagged with the wrong request's correlation ID under load. It's invisible in dev, shows up only under concurrency, and quietly lies to whoever's trying to debug with it.
Distributed tracing with spans that mean something to a human, not whatever auto-instrumentation handed you. Default OpenTelemetry auto-instrumentation gives you a span per HTTP call and per JDBC statement, which is a fine start and also not remotely what you need at 2am. What pays off is adding your own spans around the things a person would actually say out loud — "validate the batch," "apply pricing rules," "notify downstream" — tagged with whatever an on-call engineer is going to ask next (account ID, batch size, which rule set fired), not just a duration. Five well-chosen custom spans beat forty auto-generated ones nobody has time to read mid-incident.
SLOs defined on what the caller experiences, not on what the machine is doing. Success rate and latency for the actual customer-facing operation are what should page someone. CPU, heap, and thread-pool size are useful once you're already digging in, but they're a bad primary signal, because a system can blow through every user-facing SLO it has while its resource metrics stay comfortably in range — a downstream dependency degrading, a lock contention pattern, a query plan that quietly flipped after an ANALYZE. I've seen the opposite just as often, too: heap sawtoothing right up against its limit for weeks, zero customer impact, because the GC was simply doing its job.
What's cargo-culted
The flip side of all three shows up constantly. Dashboards with thirty-plus panels because a tool made it easy to add another one, not because anyone actually looks at it during an incident — you can usually spot these by asking someone what a given panel is for and watching them struggle. Alert thresholds set on infrastructure metrics because they're the easiest numbers to alert on, not because they mean anything to a customer — "CPU over 80%" paging someone at 3am for a service that's still hitting every latency target it has. And tracing bolted on because "we should probably have tracing," with nobody deciding up front what question it's supposed to answer, which is how you end up with gorgeous flame graphs that nobody opens because they don't answer whatever's actually on fire.
The cost of cargo-culted observability isn't just wasted setup time, either. It's arguably worse than having nothing, because it hands people false confidence — a green dashboard everyone's learned to trust, watching the wrong thing the whole time.
From logs to an SLO dashboard
The correlation ID buys you more than debuggability, though. It's also the join key that turns raw logs into an SLO dashboard — and doing that well really comes down to logging three events per request instead of one.
Most services log a single line per request, at the end, with an outcome and a duration. That's enough to reconstruct what happened. It's not enough to reconstruct what's currently in flight, which matters the moment you want to know whether a spike in flight requests is about to become a spike in failures. So log three distinct events instead, all carrying the same correlation ID, the API or endpoint name, and a timestamp: initiated when the request lands, then exactly one of succeeded or failed when it wraps up, with the response time attached to whichever one fires.
log.info("event=initiated api={} correlation_id={}", apiName, correlationId);
// ... on completion ...
log.info("event=succeeded api={} correlation_id={} response_time_ms={}",
apiName, correlationId, responseTimeMs);
log.info("event=failed api={} correlation_id={} response_time_ms={} reason={}",
apiName, correlationId, responseTimeMs, failureReason);
Three events instead of one sounds like extra logging for no good reason, right up until you write the query that consumes it. In Splunk, that event shape is exactly what stats and eval want to chew on, per API, over whatever window you pick:
index=app_logs sourcetype=api_events
| stats
count(eval(event="initiated")) as initiated
count(eval(event="succeeded")) as succeeded
count(eval(event="failed")) as failed
perc95(response_time_ms) as p95_response_ms
perc99(response_time_ms) as p99_response_ms
by api
| eval success_rate = round((succeeded / initiated) * 100, 2)
| eval slo_met = if(success_rate >= 99.5 AND p95_response_ms <= 300, "Met", "Breached")
One query, and you've got everything an SLO conversation actually needs, per API: a success rate pulled from real traffic instead of a sample, a latency percentile instead of an average that hides the tail, and a straight pass/fail against whatever number you've committed to. Wire a single-value panel to it with a color threshold — green on "Met," red on "Breached" — and now the dashboard just answers "are we meeting our SLO," instead of handing someone a wall of infra charts and making them do the math in their head. Trend the same query over a rolling window (daily, weekly, whatever your compliance period is) rather than a live snapshot, because a five-minute blip and a day-long breach need very different responses, and a snapshot can't tell you which one you're staring at.
Splunk actually supports this at two levels, and it's worth knowing both exist. Everything above you can build by hand in SPL against ordinary indexed logs — no special license, no separate product, just events shaped the right way. But Splunk also ships a native SLO layer built for exactly this: Splunk Observability Cloud lets you define an SLO directly as a Service Level Indicator — "proportion of requests that succeeded," say, or "proportion of requests under a latency threshold" — plus a target percentage and a compliance period, and it tracks an error budget and burn rate against that automatically, with burn-rate alerts that fire based on how fast you're eating the budget rather than one static threshold crossing once. Splunk ITSI does something similar on the core platform, rolling per-API KPIs like these into a broader service health score. Which one you reach for is a real decision — hand-rolled SPL is faster to stand up and easier to change on a whim, the native SLO objects earn their keep once you've got enough services that doing error-budget math by hand stops scaling — but either way, the capability itself is real: yes, you can define an SLO per API and see exactly how you're doing against it, not just stare at a pile of unrelated metrics and guess.
One catch worth being upfront about: this only works if api is granular enough to actually mean something. A dashboard built on one lumped success rate and one lumped p95 across every endpoint in a service can pass its SLO while its single worst, most-hammered endpoint is quietly breaching its own the entire time. That by api in the query above isn't there for style — drop it, and you've built an SLO dashboard that's technically correct and practically useless.
Verifying this actually works
The check that actually catches gaps here isn't "did the dashboard render." Kill a downstream call mid-flight on purpose, then confirm the failed event actually lands with a real response_time_ms and a populated reason — not a silently dropped event, and not one that only ever fires on the happy path because that's the only path anyone tested. Then run a load test that forces thread reuse under concurrency and grep the logs afterward for correlation IDs attached to the wrong request. Neither bug shows up in a clean-path test, and both are exactly the kind of thing that makes an SLO dashboard confidently report the wrong number during the one incident where it actually matters.
Takeaways
- A green infrastructure dashboard and a failing customer request can coexist just fine. Alert on what the caller experiences — success rate and latency for the actual operation — and treat CPU, heap, and thread-pool metrics as diagnostic, not primary.
- Structured logging only pays off with a correlation ID that survives the whole request across every service, and thread reuse means an uncleared MDC will eventually mislabel someone else's request.
- Custom spans around units of work a human would actually name beat forty auto-generated ones nobody has time to read mid-incident.
- Logging
initiated,succeeded, andfailedas three separate events, each carrying the correlation ID and response time, gives you exactly the fields an SLO query needs — success rate and latency percentile, per API, over any window you want. - Splunk supports SLO tracking at two levels: hand-built SPL over ordinary logs when you want something fast and cheap, and native SLO objects with error budgets and burn-rate alerts (Observability Cloud) or KPI-based service health scoring (ITSI) once you've got enough services that doing the math by hand stops making sense.
- An SLO measured at too coarse a grain — one number across every endpoint — can pass while your worst endpoint is breaching on its own. Break it out by API, or you're really just measuring an average and calling it an SLO.
- Test the failure path, not the clean one. Kill a call mid-flight and check that the failure event and correlation ID attribution both survive it — that's the scenario where a broken observability setup actually lies to you.
This one stayed scoped to plain request/response services on purpose. What changes once part of the system is an LLM making its own sequence of tool calls — token usage as an SLO input, latency for a call whose duration you don't control, tracing a call graph the agent invents on the fly — is enough of a shift that it deserves its own piece rather than a rushed section bolted onto this one.
