How I spotted a slow release in SigNoz using one OTel attribute

Every deploy is a small bet. You push a new version, the rollout goes green, and then you move on. If that version is 40% slower than the one before it, the usual way to find out is a customer complaining or someone happening to look at multiple dashboards at the right moment.

I spent an afternoon on a question I'd never actually sat down and answered: can I put version N and version N-1 of the same service side by side and see which one is slower?

Turns out yes, and it needs one OpenTelemetry resource attribute plus one Query Builder feature with Signoz. No custom tooling, no extra pipeline. This article is the exact setup, running locally on kind, including the bits that cost me time.

Running SigNoz on a kind cluster

My investigation was on my company's dev cluster, but since I can't share that environment, we will recreate the needed env on a local Kind cluster instead. Everything shown below was run on a fresh Kind cluster with nothing else installed.

Before moving ahead - our kind cluster should have enough resources to run Signoz. On native Linux there is no slider, because there is no VM. kind nodes are Docker containers running straight on the host kernel, and they get whatever's free. Confirm this:

kubectl describe node signoz-demo-control-plane | grep -A5 "Allocatable"
# cpu:     12
# memory:  32544504Ki

if you are on Mac/Windows and it doesnt show a minimum of 8GB RAM allocated - then you can bump the memory slider in Docker Desktop.

kubectl get storageclass

kind ships standard via the local-path provisioner, so that's what goes in the values file:

# values.yaml
global:
  storageClass: standard

clickhouse:
  installCustomStorageClass: true

helm repo add signoz https://charts.signoz.io
helm repo update

helm install signoz signoz/signoz \
  --namespace signoz --create-namespace \
  --wait \
  --timeout 1h \
  -f values.yaml

Then port-forward and confirm:

kubectl port-forward -n signoz svc/signoz 8080:8080
curl http://localhost:8080/api/v1/health

The first page is a signup screen. On a self-hosted instance that just creates a local admin account in your own database, so any email works.

Simulating two deploys without deploying anything

I didn't want to build and ship a real service twice just to test a comparison. telemetrygen, from the OpenTelemetry Collector contrib repo, generates synthetic OTLP traces with whatever resource attributes you hand it.

go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@latest

Port-forward the collector's OTLP port and send a batch:

kubectl port-forward -n signoz svc/signoz-otel-collector 4317:4317

telemetrygen traces --otlp-insecure \
  --otlp-endpoint localhost:4317 \
  --service my-test-app \
  --otlp-attributes 'service.version="v1.0.0"' \
  --traces 30

my-test-app showed up in the Services list within seconds, with telemetrygen's two default span names (lets-go and okey-dokey-0) and populated latency, rate and Apdex charts.

The most useful thing I learned all afternoon: if you send a second batch with a different service.version and nothing else changed, both versions read identically. Mine were both exactly 123 µs across p50, p90 and p99. Default telemetrygen spans are microsecond-scale and uniform, so you end up plotting the generator's own overhead rather than anything resembling a workload.

Two flags fix that. --span-duration sets how long each span claims to take, which is the latency number you're going to plot. --rate spreads generation across real time so the result is a curve rather than a dot.

# the "healthy" version
telemetrygen traces --otlp-insecure \
  --otlp-endpoint localhost:4317 \
  --service my-test-app \
  --otlp-attributes 'service.version="v1.0.0"' \
  --traces 60 --rate 1 --span-duration 100ms

# the version that regressed
telemetrygen traces --otlp-insecure \
  --otlp-endpoint localhost:4317 \
  --service my-test-app \
  --otlp-attributes 'service.version="v2.0.0"' \
  --traces 60 --rate 1 --span-duration 250ms

Now there's a real regression to go find: 100ms against 250ms.

Grouping by service.version

Here's the whole idea. service.version is a standard OpenTelemetry resource attribute, and SigNoz can group any trace aggregation by it. Set it per build in your pipeline, and every version of your service becomes its own line on a chart.

Dashboards -> New Panel -> Time Series, data source Traces:

• Filter: service.name = 'my-test-app'
• Aggregation: p99 on duration_nano
• Group By: service.version
• Legend format: service.version = {{service.version}}

The legend format field is the one I'd have skipped and regretted. Without it, grouped series get generic labels and you can't tell which curve is which. {{attribute_name}} placeholders pull the actual value into the legend, so your lines are labelled v1.0.0 and v2.0.0 directly.

Stage & Run Query, and there they are — two labelled curves, one settling around 100ms and one around 250ms.

One formatting step worth doing straight away: the y-axis reads raw nanoseconds by default, so a 250ms p99 renders as 250000000 and the chart is hard to read at a glance. Panel Settings -> Formatting & Units -> set the unit for the y-axis, and it renders as ms.

The time axis stops mattering here, which is the part I find genuinely useful. It doesn't matter that v1 ran an hour ago and v2 ran five minutes ago. They're grouped by version, so you're comparing versions, not time windows.

A few extensions of the same pattern, once you're on a real service instead of a synthetic one:

• Group error rate by version too. Same query shape, count() aggregation with a status_code filter, grouped by service.version. A release can hold latency perfectly steady while quietly doubling its 5xx rate, and the latency panel will never show you that.
• Group by service.version alongside k8s.pod.name during a rolling update, when only some pods are running the new version. That tells you whether you're looking at a code regression or a half-finished rollout.
• Save the panel as a view. The Trace Explorer's bottom bar has Save this view, so the version comparison becomes one click instead of a query you rebuild each time you're suspicious.

Alerting on it

Once the query exists, turning it into an alert is quick. Alerts -> New Alert -> Trace-based Alert, then the same filter and p99(duration_nano) aggregation, with a threshold sitting above your known-good baseline. You can keep service.version in the Group By, so each version is evaluated as its own series rather than blended into one number.

For my test data a threshold of 150ms sits neatly between the two versions, so v1.0.0 stays quiet and v2.0.0 crosses it.

What I'd tell myself before starting

service.version is the highest-leverage attribute in this whole setup, and it's the one most people leave unset or hardcoded to something meaningless. Wire it to your git SHA or release tag in the build. Everything above depends on it and it costs one line in a Dockerfile or collector config.

Second: when synthetic data shows no difference between two things you know are different, suspect your generator before you suspect the tool. --span-duration was the difference between a flat, useless chart and a real one.

What's next

Grouping by version gives you an honest answer to "did this release make things worse", and it works today with nothing more than an attribute and one panel.

The piece I want next is the automation around it: something that notices a new version went live, runs this comparison on its own, and tells me the answer without me having to suspect anything first. That's what I'm building next week, and I'll write it up the same way.

If that's interesting, I post this kind of thing regularly:

• GitHub: 1Shubham7
• X: @1shubham7_
• YouTube: @1shubham_7
• LinkedIn: 1shubham7

Docs I leaned on: SigNoz Query Builder · telemetrygen README · OpenTelemetry resource semantic conventions