Errors spiked at 14:07 and the graphs made no sense. Success rate dropped to about 70%, not zero. Some requests succeeded, some returned 500 with a Postgres connection timeout, and a few completed normally. Our dashboard showed three healthy instances the entire time. Every probe had passed. The load balancer was still routing traffic to an app that had been unable to talk to its database for twenty minutes.
That is the failure mode. A health check that only proves the process exists keeps a broken instance in rotation, and the outage shows up as random errors instead of a clean removal.
Liveness and readiness answer different questions
Liveness asks: should the supervisor restart this process? Readiness asks: should the load balancer send it traffic? If one endpoint answers both, you get one of two bad outcomes.
When the check is shallow, it tells the supervisor "no restart needed" โ correct, the process really is fine. Then it tells the load balancer the same thing, and that answer is wrong. The binary answer cannot be right for both readers.
When the check is deep and shared, a dead database makes every instance unready, so every instance gets pulled, and the load balancer has nowhere to send traffic. That is a cascading failure dressed up as a health check.
Keep them separate. This is a Kubernetes example but the split applies to any orchestrator or proxy.
livenessProbe:
httpGet:
path: /livez
port: 8080
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /readyz
port: 8080
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2
/livez returns 200 if the event loop can serve an HTTP response. Nothing else. It never queries the database, never calls a downstream service, and never allocates a connection pool.
/readyz checks the things that must work for a request to succeed. It is allowed to fail, and failing means removal from rotation, not a restart.
The first time I split these apart, the restart storms stopped. The app had been OOM-restarting because the deep check ran on every liveness tick and opened database connections the collector never released. A probe that does real work has real cost, and paying that cost every ten seconds on every instance adds up.
Timeouts are the whole check
A readiness probe with no timeout is worse than no probe. If the database call hangs, the probe hangs, the orchestrator waits, and traffic keeps flowing to the stuck instance for the entire timeout window.
Give the check a hard deadline shorter than the probe's own timeout, and fail closed:
import asyncpg
async def readyz(pool):
try:
async with pool.acquire(timeout=0.5) as conn:
await conn.fetchval("SELECT 1", timeout=0.5)
return 200, {"status": "ready"}
except (asyncpg.PostgresError, TimeoutError) as exc:
return 503, {"status": "unready", "reason": type(exc).__name__}
Measure what your database actually returns under normal load โ p99 of SELECT 1 on a warm pool โ then set the probe deadline above that and far below your request timeout. Do not guess the number. Log the probe duration and read the histogram.
Cache the result too. If the load balancer hits /readyz from ten addresses per second, do not run ten queries per second. Cache the verdict for one or two seconds. The cache TTL becomes your detection latency, so keep it short, and document it.
Return 503, not 200 with a JSON body that says "status": "degraded". Load balancers read status codes. A 200 with bad news inside is a 200.
Degraded is not down
The hardest case is a dependency that is slow, not dead. The database accepts connections but queries take four seconds instead of forty milliseconds. Marking the instance unready here removes capacity exactly when the system needs all of it, and the next instance inherits the same slow database.
Shed load instead of failing. Keep /readyz at 200, and expose a second signal the application itself uses:
func (a *App) admit(ctx context.Context) error {
if a.db.Latency() > a.degradedThreshold {
if atomic.LoadInt64(&a.inflight) > a.maxInflight/2 {
return ErrShedLoad
}
}
return nil
}
Return 429 or 503 with Retry-After on the shed requests. The instance stays in rotation, serves what it can, and stops piling work onto a dependency that is already behind. Readiness stays green. The error rate is visible and bounded instead of total.
The process being up is not a health signal
An HTTP 200 on /health tells you a socket accepted a connection and a handler returned. It says nothing about the database, the cache, the queue, or the disk. It is a liveness signal at best, and most teams already get liveness for free from the orchestrator's process supervisor.
Before you trust the next green dashboard, do this: kill the network path from one instance to its database, then watch. If traffic keeps arriving, your health check is lying. If every instance drops out at once, it is lying in the other direction.
I write about production failures in Postgres, queues, and distributed systems.
Subscribe by email ยท RSS ยท Bluesky
