Exposing a /metrics endpoint takes five minutes. Getting the instrumentation itself right — metrics that stay cheap at scale, catch real problems, and don't lie under load. Here's a breakdown of the design decisions behind a recent metrics layer built for a Go API.
RED as the baseline, not the ceiling
Request rate, error ratio, and latency:
-
http_requests_total{method, path, status}for volume and error breakdown -
http_request_duration_secondsas a histogram, buckets tuned to the actual latency distribution -
http_requests_in_flightas a gauge for concurrency pressure -
http_response_size_bytesto catch payload bloat before it manifests as latency
The label that matters most here is path. Using the resolved URL instead of the route template turns every /users/42, /users/43... into its own series — cardinality explodes on nothing but normal traffic, and unmatched routes (scanner noise, bad paths) make it worse. Route templates as labels, unmatched paths collapsed into a single bucket, and the metrics endpoint itself excluded from the middleware so scrape traffic doesn't pollute its own numbers.
Connection pool saturation
Database exhaustion rarely announces itself early through query timing alone. Wiring NewDBStatsCollector against the connection pool surfaces open/in-use/idle counts plus wait count and wait duration — the earliest indicator that a pool is under pressure, well before it shows up as user-facing latency.
That instrumentation only means something paired with actual limits: max open connections, idle connection caps, connection lifetime. Without bounds, a spike exhausts the database directly. With them, the same spike shows up as queuing in the metrics — visible, diagnosable, alertable.
Security events belong in the metrics pipeline
Structured logs answer "what happened." They don't answer "is this happening right now, at what rate." Auth failures, failed logins by reason, account lockouts, rate-limit triggers, and forbidden-access attempts got a parallel path into Prometheus — security_events_total{event, reason}, logins_total{result} — incremented alongside the existing log calls.
That's the difference between discovering a brute-force pattern by grepping logs after the fact, and alerting on the rate the moment it starts climbing.
Reliability and runtime signals
-
panics_recovered_totalfrom the recovery middleware — stability issues that would otherwise blend into a pile of 500s -
cache_operations_total{cache, result}on hit/miss paths — actual numbers about cache effectiveness -
app_build_info{service, version, env}— correlating any metric back to the exact build and environment it came from - Default Go collectors, registered once: goroutines, memstats, GC pauses, CPU, resident memory — no custom code, and usually the first sign of a leak long before it's user-facing
The principle underneath it
Every metric is a standing commitment — a series that has to be maintained — and every label is a multiplier on its cost. The goal was never maximum coverage. It was a deliberately small set of signals that answer "is this healthy, and if not, why" without opening a log file first. The mechanics of Prometheus are the easy part. Choosing what deserves to be measured, and building the guardrails that keep it cheap as traffic grows, is the actual work.
