The CoreDNS Black Hole: how one dead DNS pod broke our API gateway

Our API gateway started handing this back to roughly a third of all requests:

{"source":"auth","message":"Upstream Service Unavailable","error":"Upstream Service unreachable"}

Restart the gateway and everything is fine. A day later, it's back. No deploy, no code change, no traffic spike, nothing in the backend logs — the backends were healthy the whole time.

The cause was one open network socket pointing at a pod that no longer existed. The mechanism is genuinely strange, so let's start with an analogy.

The analogy: the receptionist who went home

Your gateway is a clerk at a busy counter. Before serving anyone, the clerk needs a phone number — "which desk handles billing today?" — so it calls the office directory.

A sensible clerk dials the directory's main switchboard every time. Our clerk did something subtly different: it dialled once, got through to Priya at the directory, and then kept that line open forever, because opening a line costs time and Priya always answers.

One night the building is reorganised. Priya's desk is removed. Her extension now goes nowhere.

Next morning the clerk picks up the same line, asks its question… and hears nothing. It waits. Still nothing. It gives up, tells the customer "billing is unavailable" — and then, crucially, asks the very same dead line again for the next customer. And the next. Forever.

Why doesn't it redial? Because the clerk only redials when the phone reports "this line is disconnected." Silence isn't an error. Silence is just silence, so it waits again.

That's the whole bug. Now swap in the real names:

If you haven't met these three before

CoreDNS is the cluster's phone book. In Kubernetes you don't address a pod by IP — you address a Service by name, like iamplatform.my-ns.svc.cluster.local. CoreDNS is the (usually two-pod) deployment that turns those names into IPs, and essentially everything in the cluster asks it, constantly.

Cilium is the networking layer. It does load-balancing inside the kernel using eBPF, instead of the older kube-proxy/iptables approach. The detail that matters here: when your app calls connect() on a Service IP, Cilium rewrites the destination there and then to one specific backend pod's IP. That's part of what makes it fast — and it means your socket is attached to that one pod, not to the Service as a whole.

Karpenter is the node autoscaler. It adds nodes when pods need room and — the part that matters here — removes under-used ones to save money. Pods on a removed node get rescheduled elsewhere, CoreDNS pods included. Completely routine behaviour.

Three reasonable components, each doing its job, combining into a permanent black hole.

_{One socket, glued to one pod. When Karpenter replaces that pod, the socket keeps pointing at nothing.}

DNS lookup times out → nginx can't find the backend → 502 → our error handler turns that into the 503 the user sees. The backend was never even contacted.

Why it looked random

Each nginx worker has its own socket, so only the worker with the dead socket should fail — maybe 1/8 of traffic. We were seeing far more than that.

The reason was one line of config: accept_mutex off with no reuseport. Without reuseport, workers race to accept new connections and one worker reliably wins most of them. If that greedy worker is the one holding the dead socket, one broken worker is effectively the whole pod.

_{Without reuseport one worker wins most connections — so one dead socket looks like a dead pod.}

That also explains the "intermittent" feel: whether a pod looked healthy or dead depended on which worker happened to be the greedy one.

We proved it instead of guessing

The temptation here is to ship a plausible fix and hope. Instead we wrote down predictions first — which pods would break, which worker in each, and roughly what failure rate — then deliberately deleted a live CoreDNS pod and checked.

The predictions matched. That mattered, because the first proposed fix (tune kernel socket settings, add more replicas) targeted a completely different theory — port exhaustion — that the evidence ruled out. Without the experiment we'd have shipped it and stayed broken.

Reading the sockets needs no special tooling:

# which CoreDNS pod is each worker's DNS socket glued to?
cat /proc/net/udp     # decode the hex peer address column

The fix: three layers

_{Failure rate during a deliberate CoreDNS pod deletion, at each layer of the fix.}

1. Spread the traffic — listen 8080 reuseport. The kernel deals new connections evenly across workers, so a broken worker costs 1/N of a pod instead of all of it. This caps the damage; it doesn't prevent it.

2. Keep a spare line — list the same DNS name twice in the resolver directive. nginx opens one socket per listed address, so two entries means two independent sockets landing on (probably) two different CoreDNS pods. You must also raise resolver_timeout above nginx's hard-coded ~5s retry interval, or it never gets around to trying the second socket:

resolver  kube-dns.kube-system.svc.cluster.local
          kube-dns.kube-system.svc.cluster.local  valid=60s ipv6=off;
resolver_timeout 10s;   # was 3s — too short to ever reach socket #2

Honest cost: when the first socket is dead, requests wait ~5 seconds before the retry succeeds. That's a real, user-visible stall. This layer masks the problem, it doesn't heal it.

3. Take DNS off the request path entirely — the actual cure. Instead of looking up the address while a request waits, declare each backend as an upstream group that nginx resolves in the background and pools connections to (nginx 1.27.3+ in OSS; pooling explained two sections down, if the term is new):

upstream mt_iamplatform {
  zone   mt_upstreams 2m;
  server iamplatform.my-ns.svc.cluster.local:22036 resolve;  # background refresh
  keepalive 32;
}

The magic property: if a background refresh times out, nginx keeps the last known good address instead of failing the request. A DNS outage stops being an outage.

The numbers

Same experiment each time — delete a live CoreDNS pod:

In the last run a worker was still holding a dead DNS socket the whole time. Nobody noticed, which is the point.

The rule we refused to break: nginx must always boot

Worth a detour, because it's the reason the fix looks the way it does.

If you name a backend literally — proxy_pass http://iamplatform.my-ns.svc.cluster.local:22036;, or a plain server line inside an upstream — nginx resolves that name once, at config load. Efficient. But if any one of those names fails to resolve — a service not created yet, a namespace being rebuilt, a typo in a new route — nginx prints [emerg] host not found in upstream and refuses to start at all. One missing backend takes down all ~50 routes, including the 49 that were fine.

So our routes deliberately use a variable:

set        $backend "iamplatform.my-ns.svc.cluster.local:22036";
proxy_pass http://$backend;        # a variable -> resolved per request

nginx can't resolve a variable at config load, so it doesn't try. The gateway boots knowing nothing about its backends and looks each one up at request time. A missing service now returns 502 on its own route, and nothing else notices.

_{A literal hostname makes every route depend on every backend at boot. A variable keeps the failure local.}

Layer 3 had to preserve that. The obvious way to add pooling is a hard-coded upstream block — which hands straight back the boot-time dependency we'd avoided. Two details kept it safe:

• server ... resolve tells nginx to skip the config-load lookup entirely and resolve in the background instead, so an unresolvable backend still can't block startup.
• $mt_backend holds the upstream group's name, not a hostname — so proxy_pass is still pointed at a variable and never quietly became a literal.

set        $backend "mt_iamplatform";   # the group's name, not a host
proxy_pass http://$backend;             # this line never changed

A gotcha if you try this: nginx only matches a variable proxy_pass to an upstream group when the value contains no port. mt_iamplatform resolves to the group; mt_iamplatform:22036 is treated as a hostname and fails.

So we gained pooling and background DNS and kept "a broken backend is that backend's problem."

Connection pooling, in 60 seconds

Both the fix above and the bug below turn on this, so here's the short version.

By default, when nginx proxies a request it opens a fresh TCP connection to the backend, sends the request, reads the response, and closes it. Every request pays a three-way handshake, and every close parks a socket in TIME_WAIT on nginx for 60 seconds, holding one of ~28,000 ephemeral ports. At a few hundred requests a second, that adds up.

Pooling — nginx calls it upstream keep-alive — means: don't close it. Keep the connection and let the next request to the same backend ride the line that's already open.

_{Without pooling, every request pays a handshake and leaves a socket in TIME_WAIT. With it, one connection carries many requests.}

Switching it on is three things, not one, and missing any of them leaves you with a pool that silently never engages:

upstream my_backend {
    server backend.my-ns.svc.cluster.local:8080;
    keepalive 32;              # how many idle connections to cache
    keepalive_requests 1000;   # reuse one this many times, then close it
    keepalive_timeout 60s;     # close it after this long idle
}

location /thing {
    proxy_pass http://my_backend;
    proxy_http_version 1.1;           # keep-alive needs HTTP/1.1
    proxy_set_header Connection "";   # don't forward the client's "Connection: close"
}

Four things that catch people out:

1. keepalive 32 is per worker process, not per pod. Sixteen workers means up to 512 idle connections from a single pod.
2. You need a named upstream block. A bare proxy_pass http://host:port; cannot pool at all.
3. Whoever closes first matters. Your keepalive_timeout has to sit below the backend's own idle timeout, or you'll eventually pick up a connection just as the backend's FIN arrives — upstream prematurely closed connection.
4. nginx only pools a connection whose response it finished reading. That one is a trapdoor, and it's the one we fell through.

Checking which you've got needs no extra tooling — count sockets to the backend from inside the pod:

# column 4 is the TCP state in hex: 01 = ESTABLISHED, 06 = TIME_WAIT
awk '$4=="01"' /proc/net/tcp | wc -l
awk '$4=="06"' /proc/net/tcp | wc -l

A healthy pool is a steady handful of ESTABLISHED and almost nothing in TIME_WAIT.

The bonus bug: a pool that never pooled

Measuring after a fix is how you find the next one. With DNS off the request path we looked at the gateway's sockets again: ~13,000 TIME_WAIT sockets per pod and zero pooled connections to our auth service — even though the config had declared keepalive 64 for it for years.

Back to the clerk. Before serving anyone, it also has to ask the auth desk "is this person allowed in?" The clerk only needs the answer written on the envelope — yes/no, and who you are. But the auth desk kept stuffing a full brochure inside every reply. And the clerk has a rule: I can only keep the phone line open if the reply is envelope-only. So it hung up after every call and redialled for the next customer.

In nginx terms: auth_request marks its subrequest header-only and never reads the response body — and nginx returns the connection to its keep-alive pool only when the response has no body (204, or a non-chunked Content-Length: 0). Our auth service answered with JSON, so the pool could never engage. Not just untidy: at ~470 connections per second to one backend pod you run out of ephemeral ports, and during an auth-service rollout — when there may be only one pod left — a busy gateway sits uncomfortably close to that.

The fix belonged in the auth service. But that service has its own ingress and other consumers who genuinely want the body, so stripping it for everyone would break them.

Tag the caller, not the endpoint

The gateway already stamps every auth subrequest with a header:

proxy_set_header X-Sent-From "nginx-pods";

So the auth service now drops only the body — never the status code, never the headers — for requests carrying that tag:

// middlewares.HeaderOnlyForGateway, wired on POST /auth
if !strings.EqualFold(strings.TrimSpace(c.GetHeader("X-Sent-From")), "nginx-pods") {
    c.Next()   // every other caller: unchanged, body and all
    return
}
c.Writer = &headerOnlyWriter{ResponseWriter: c.Writer}  // keep status + headers,
c.Next()                                                // discard the body

Nothing is lost, because the gateway reads only the response headers (X-Identity-*, X-Session-ID, X-Auth-Error-*) and builds its own client-facing JSON from them. Handlers keep calling c.JSON(...) exactly as before — they don't know the middleware exists.

_{A header tags the caller, so one shared endpoint can answer the gateway and everyone else differently.}

One easy-to-miss detail: once pooling works, the two sides' idle timeouts must be ordered. Go's IdleTimeout is set to 120s, deliberately above the gateway's keepalive_timeout 60s, so nginx is always the side that closes an idle connection. If the server closed first, the gateway would sometimes pick one up just as its FIN arrived — surfacing as upstream prematurely closed connection.

Takeaways

1. A long-lived connection is a cached decision. Anything you connect to once and keep — DNS sockets, pooled connections, resolved IPs — is a bet that the thing on the other end won't be replaced. In Kubernetes, it will be.
2. Timeout ≠ error. Many clients recover from a refused connection and hang forever on silence. Ask which one your retry logic keys on.
3. Write your predictions down before you break something. It's the cheapest way to tell a real diagnosis from a plausible story — and it's what stopped us shipping the wrong fix.
4. Fail small, not early. Resolving every dependency up front feels safer, but it couples every route to every backend. Late binding — looking things up when you actually need them — is what keeps one missing service from becoming a full outage.

The gateway is nginx on Kubernetes with Cilium and Karpenter, but the shape of this bug isn't nginx-specific. Any client that holds one socket to one replica of a service has it.