The Golden Signals: A Practical Implementation Guide

Four Metrics to Rule Them All

Google's SRE book introduced the four golden signals: Latency, Traffic, Errors, and Saturation. Simple concept, but I've seen teams struggle with implementation.

Here's a practical guide from someone who's implemented them across 50+ services.

Signal 1: Latency

Not all latency is equal. You need to track successful requests and error requests separately.

# Bad: Average latency
latency = total_request_time / total_requests  # Useless

# Good: Percentile latency, separated by status
from prometheus_client import Histogram

REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'Request latency',
    ['method', 'endpoint', 'status_class'],
    buckets=[.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
)

@app.middleware
async def track_latency(request, call_next):
    start = time.time()
    response = await call_next(request)
    duration = time.time() - start
    status_class = f"{response.status_code // 100}xx"
    REQUEST_LATENCY.labels(
        method=request.method,
        endpoint=request.url.path,
        status_class=status_class
    ).observe(duration)
    return response

Alert on p99, not p50. Your happiest users don't need help.

- alert: HighLatencyP99
  expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5
  for: 5m
  labels:
    severity: warning

Signal 2: Traffic

Traffic tells you "is this normal?" It's the context for every other signal.

# Current request rate
rate(http_requests_total[5m])

# Compare to same time last week
rate(http_requests_total[5m]) 
  / 
rate(http_requests_total[5m] offset 7d)

# Alert on sudden drops (possible outage nobody noticed)
- alert: TrafficDrop
  expr: >
    rate(http_requests_total[5m]) 
    < 
    (rate(http_requests_total[5m] offset 1h) * 0.5)
  for: 10m
  annotations:
    summary: "Traffic dropped >50% compared to 1 hour ago"

Traffic drops are often more concerning than traffic spikes.

Signal 3: Errors

Track error rate as a percentage, not absolute count:

# Error rate percentage
(
  sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total[5m]))
) * 100

But also track error types separately:

error_categories:
  - 5xx: "Server errors (our fault)"
  - 4xx_excluding_404: "Client errors (possible API issue)"
  - timeout: "Request timeouts"
  - circuit_breaker: "Dependency failures"

Signal 4: Saturation

The most underrated signal. Saturation answers: "how close are we to full?"

# CPU saturation
process_cpu_seconds_total / container_spec_cpu_quota

# Memory saturation
container_memory_working_set_bytes / container_spec_memory_limit_bytes

# Connection pool saturation
active_connections / max_connections

# Queue saturation (the one everyone forgets)
message_queue_depth / message_queue_capacity

Alert before you hit 100%. I use 80% as the threshold for warning and 95% for critical.

Putting It All Together

Every service gets a standard dashboard with four rows:

Row 1: Latency   [p50] [p90] [p99] [error latency]
Row 2: Traffic    [rate] [vs last week] [by endpoint]
Row 3: Errors     [rate %] [by type] [by endpoint]
Row 4: Saturation [CPU] [Memory] [Connections] [Queue]

This fits on one screen. No scrolling. Any engineer can assess service health in 10 seconds.

The Anti-Pattern

Don't build a golden signals dashboard per service manually. Template it:

{
  "dashboard": {
    "title": "Golden Signals: {{ service_name }}",
    "templating": {
      "list": [
        { "name": "service", "type": "query" },
        { "name": "environment", "type": "custom", "options": ["prod", "staging"] }
      ]
    }
  }
}