Our checkout handler called three downstream services in sequence, and each call had a thirty-second timeout copied from an old Stack Overflow answer. The load balancer gave up after thirty seconds. The handler could legally spend ninety. Requests piled up, goroutines stacked on the stalled calls, and the whole API went down because one read replica started fsyncing slowly. Everything stayed "healthy" on the health checks.
I found it in the metrics. Response time per handler looked fine at the p50. It was the queue depth and the goroutine count that told the story.
The real bug was that our timeout was not a number. It was infinity.
Most services ship with a default timeout that is either infinite or one constant applied everywhere. Go's http.Client has no timeout unless you set one. Python's requests has none either. Java's HttpURLConnection treats a zero read timeout as never. The pool does not grow because someone forgot a line. It grows because the absence of a line means "wait forever," and forever is longer than any request should live. That failure does not stay local. It climbs: the dependency stalls, your threads block, your caller's threads block, and the outage escalates one hop per timeout period.
The three numbers
Connect timeout: how long to establish the TCP connection before giving up. This catches unroutable hosts, dead load balancers, and half-open sockets.
Request or read timeout: how long to wait for a response after you are connected. This catches a dependency that accepts the connection and then hangs.
Total deadline: the entire budget for the call, including connection, reading, retries, and any queueing you do. Set it once and propagate it, rather than resetting a timer at every layer.
The contract is arithmetic. The sum of the downstream budgets must be smaller than your own deadline. If your caller allows two seconds and you give each of three dependencies two seconds, you have no contract. You have an unbounded queue with extra steps.
In Go I name all three on purpose. net.Dialer.Timeout handles connect. http.Client.Timeout covers the whole request, so it is often the deadline, which is why I prefer to derive a context and let it rule both.
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return err
}
resp, err := client.Do(req) // client.Timeout as a backstop
if err != nil {
// context.DeadlineExceeded or a timeout error: the dependency is slow
return err
}
Infinite defaults are the bug
An infinite default is a promise you made to nobody. It converts a slow dependency into thread exhaustion upstream, because the caller cannot cancel work that has no end. Once I started setting finite timeouts, the failure mode changed from "everything is down" to "one call returns an error and a circuit breaker opens." That is a much better afternoon.
Health checks do not save you here. A process can answer /healthz in a millisecond while every worker is parked on a socket that will never answer.
Deadlines must propagate
Cancellation only works if it travels with the request. A fresh context.Background() in the middle of the call chain is where the deadline dies.
In Go, derive every nested context from the incoming one and never call Background inside a request path. Read the remaining time with ctx.Deadline() and fail fast if it is already near zero.
import httpx, asyncio
timeout = httpx.Timeout(connect=0.3, read=1.5, write=0.5, pool=0.2)
async with asyncio.timeout(2.0):
async with httpx.AsyncClient(timeout=timeout) as client:
return await client.get(url)
asyncio.timeout wraps the whole block, so the outer deadline wins even if the client-level read timeout is longer. If you cross a process boundary, pass the deadline too: an X-Request-Deadline header the callee converts back into its own context, or the gRPC grpc-timeout metadata plus ctx.Deadline() when you build the outgoing context. A header carries no cancellation, only a timestamp, so keep those clocks close and treat an expired deadline as a refusal to start work.
Pick the numbers from measurement
I stopped guessing after the checkout incident. Pull the p99 and p99.9 latency for each downstream call from your metrics store over a normal week, then set the timeout slightly above p99.9 so real traffic passes and genuinely stuck connections die. Sum every downstream timeout plus your retry budget and confirm the total is less than the caller's deadline. Alert on timeout counters, because a timeout that fires is information about a slow dependency, not an invitation to raise the constant.
The three numbers are connect, request, and total. The one rule is that downstream budgets sum to less than the caller's own. Write them down, propagate the deadline, and measure instead of copying a number from a blog post β including this one.
I write about production failures in Postgres, queues, and distributed systems.
Subscribe by email Β· RSS Β· Bluesky
