I've been building something I call the Backend Performance Lab: a repo full of APIs that are broken in very specific, very deliberate ways. Connection pool exhaustion here. N+1 queries there. Bad pagination in a corner somewhere, waiting for someone to hit it with 200 concurrent users.
The goal isn't to demo how to fix slow code. It's to actually sit with a slow endpoint, figure out why it's slow, and write down the whole investigation — not just the fix.
This post is about the first scenario I built, a "Contract Approval" endpoint, and how I went from staring at scattered console logs to actually watching a request move through the system in real time using OpenTelemetry and SigNoz.
The Backend Performance Lab: a set of intentionally broken backend scenarios, each one built to reproduce a real production bottleneck, get benchmarked, get investigated, and get written up.
The endpoint that logs couldn't explain
The Contract Approval endpoint isn't complicated on paper. One request, wrapped in a database transaction, does roughly seven things: reads the contract, validates approval rules, writes an approval record, generates a PDF, extracts metadata from it, fires off an email notification, and pings an external ERP system before committing.
Each of those steps logged its own "done" message. Prisma printed every SQL statement it ran. I even had a little query counter tallying things up per request. On paper, I had plenty of information.
What I didn't have was a story. I had a pile of log lines with timestamps I had to manually line up in my head to figure out what happened in what order, and how long any of it actually took relative to everything else. That's fine for five operations. It stops being fine once you're chasing a performance bug across a transaction, a PDF renderer, an email queue, and a third-party integration.
Before: requests went client → Express → Prisma → PostgreSQL, and logs told me code had run, but not how the request actually behaved end to end.
Wiring up OpenTelemetry (and why auto-instrumentation wasn't the whole answer)
I added OpenTelemetry with a shared instrumentation setup I could reuse across every scenario in the lab, not just this one. The auto-instrumentation part was almost anticlimactic — turn it on, and HTTP requests, Express middleware, Postgres queries, TCP connections, and DNS lookups all started showing up in SigNoz with basically no work on my part.
But that gave me infrastructure detail, not business detail. A span called pg.query telling me an INSERT ran against the Approval table doesn't tell me why — it doesn't tell me that this was the "create approval" step of a contract workflow.
So I added spans manually around the operations that actually matter to the business logic:
return tracer.startActiveSpan(
"Approve Contract",
async (span) => {
// ...
}
)
I did this for seven boundaries: the HTTP request itself, contract approval, the database transaction, PDF generation, metadata extraction, email notification, and the ERP call. Everything else — helper functions, small utilities — I left alone. More on why below.
After: every meaningful step of the workflow, plus every database call, lands in a single trace I can actually look at.
Benchmarking first, tracing second
Before I touched any code, I wanted a number, not a feeling. I ran the endpoint under concurrent load with JMeter to get a baseline.
JMeter established the performance baseline by revealing high response times and a 57.29% error rate under load. It quantified the problem, but tracing was needed to identify where the time was actually being spent.
That answered one question: yes, the endpoint gets slow under concurrency. It said nothing about where the time was going. So I opened one of the slow requests as a trace in SigNoz instead of adding yet another log line.
The waterfall view — one request, start to finish, in the order things actually happened.
This was the first time I could see the entire lifecycle of one request as a single timeline: HTTP request in, database transaction, business validation, PDF generation, metadata extraction, email, dashboard update, ERP call, commit — laid out left to right instead of scattered across a terminal scrollback.
The flame graph made the expensive operations obvious at a glance.
And the flame graph is where it actually paid off. PDF generation, metadata extraction, and the email send were eating most of the request's duration — not the database transaction I'd originally suspected. That's the kind of thing that's genuinely hard to see from logs, because logs don't show you proportion. They show you that something happened, not how much of the total time it consumed.
What was hiding inside the trace
Because Postgres was auto-instrumented, every SQL statement showed up as a child span with its own duration and the actual query text attached.
Every query, in context, with no manual log-matching required.
And the custom spans I'd added carried business context — contract ID, workflow name, scenario name, total query count — which turned out to matter more than I expected. It's one thing to see that a span took 800ms. It's another to see that span attached to contract #4521 in the "connection pool exhaustion" scenario, which makes it trivial to filter and compare across runs.
Business context attached directly to the span, not reconstructed from a log line afterward.
At this point I wasn't just looking at telemetry. I was looking at the actual shape of a business transaction — which is a different thing than a stack of infrastructure metrics.
What actually surprised me, coming from a logging-only background
My prior observability experience was Winston for logs, Filebeat to ship them, Kibana to search them. It worked, but investigating one request meant hopping between log files and timestamps and rebuilding the timeline by hand. A few things about tracing genuinely caught me off guard.
I never touched a correlation ID. In the logging world, keeping related messages together means generating a request ID and threading it through every controller, service, and repository — and if you forget it in one place, the chain breaks. With OpenTelemetry, context propagation is automatic. Every span for a request lands under the same trace without me wiring anything.
Auto-instrumentation got me 80% of the way in about ten minutes. I expected to spend an afternoon configuring instrumentation library by library. Instead HTTP, Express, and Postgres just showed up. The actual work was deciding which business operations deserved their own custom spans — which was a much better use of my time than fighting config files.
Deciding where not to add a span was the real skill. My first instinct was to instrument everything — every helper, every small function. I tried it briefly and the resulting trace was unreadable, dozens of tiny spans burying the operations that actually mattered. Scoping spans to business boundaries (approve contract, generate PDF, notify ERP) instead of implementation details is what made the traces actually useful during an investigation. This is the part that isn't really explained well in the docs — it's something you only learn by making a trace too noisy once.
Benchmarking and tracing aren't competing tools. JMeter told me the endpoint was slow. SigNoz told me why. I don't think I would've gotten to "why" nearly as fast with logs alone, and JMeter's numbers gave the trace something to be measured against — without a baseline, "PDF generation took 900ms" doesn't mean much on its own.
Logs vs. traces, for what it's worth
Logs are still useful — they're cheap, they're simple, and for a single-service app with low complexity they're often enough. Where they fall over is exactly the case above: a request that fans out across a transaction, a rendering step, an email service, and a third-party API. At that point you're not debugging code anymore, you're debugging a timeline, and logs were never built to show you a timeline.
Where this goes next
This was one scenario. The Backend Performance Lab also has N+1 queries, missing indexes, offset pagination that falls apart at scale, Redis caching, CPU-bound endpoints, and oversized payload responses waiting to get the same treatment — benchmark it, trace it, write down what actually happened.
If you've been leaning on logs the way I was, it's worth pointing OpenTelemetry at one real workflow in something you're building, even locally. You don't need production traffic to get value out of it — half of what surprised me here happened on my laptop, not in a deployed environment.
The short version: I wasn't missing logs. I was missing a way to see the request as one thing instead of twelve.
GitHub Repository: https://github.com/mohandotdev/backend-performance-lab
