Requests that fail after a Kubernetes rollout has finished

At the start of October I measured a Kubernetes rolling update on DigitalOcean's managed Kubernetes and found that maxUnavailable: 0 still drops requests (that post is here). The clients there held keep-alive connections, and I wrote that a load generator opening a fresh connection per request "will hide most of this".

I hadn't tested that. So this time I ran the same kind of rollout with both client types side by side, on six different pod setups, and counted every failed request. Fresh connections hid nothing. They failed in a different way, and in some setups they kept failing for almost two minutes after kubectl rollout status said the rollout was done.

The setup

• kind v0.33.0 with Kubernetes v1.37.0: one control-plane node and two workers, in Docker Desktop on an ARM Mac. kube-proxy in its default iptables mode.
• The app: a 60-line Python HTTP server. Each request takes 50 ms. It waits 2 seconds after starting before it listens, to stand in for a real app's startup.
• A Deployment with 3 replicas and the default rolling update strategy, which at 3 replicas means one extra pod at a time and none taken away early.
• The client: a separate pod with 10 threads sending requests to the Service's ClusterIP for 50 seconds, with a 3-second timeout. About 5 seconds in, I changed an environment variable on the Deployment, which starts a rollout.
• Two client modes. One opens a new connection for every request, like many HTTP clients do by default. The other keeps connections open and reuses them.

Every scenario ran twice, and I show both numbers. The six pod setups:

1. naive: no readiness probe, and the app exits immediately on SIGTERM
2. readiness: a readiness probe on /healthz, checking every second, and the app exits immediately on SIGTERM
3. readiness + drain: on SIGTERM the app stops accepting connections, sends Connection: close on the requests it still has, and exits once they finish
4. readiness + preStop: a preStop hook that sleeps 5 seconds, then the app exits immediately
5. readiness + preStop + drain: both
6. readiness, no handler: the app installs no SIGTERM handler at all

Keep-alive clients: the October result again

Each run sent between 5,100 and 5,500 requests. This matches what I found on the real cluster. An open connection keeps going to the same pod until one side closes it, so the preStop sleep alone left 11 connections to be reset when the app exited. Only preStop plus a drain that closes keep-alive connections was clean. Keep the "no handler" row in mind, I'll come back to it.

New connection per request

Each run sent between 7,100 and 9,500 requests:

"After the rollout finished" means the request started more than 5 seconds after kubectl rollout status returned.

So fresh connections failed about as often as keep-alive ones in most setups, not less. The pattern was different. The drain, which fixed almost everything for keep-alive clients, did nothing here. The preStop sleep, which wasn't enough for keep-alive clients, fixed everything.

That follows from the race the October post measured at about half a second. When a pod is deleted, the kubelet sends SIGTERM and, at the same time, the pod is taken out of the EndpointSlice and kube-proxy on every node rewrites its rules. If the app stops listening the moment it gets SIGTERM, new connections still arrive for a short while. While the pod's network exists they're refused. Once it's gone, the SYN goes nowhere and the client waits for its timeout. Most of the failures here were those timeouts. A drain doesn't help, because stopping the listener is the first thing it does. The preStop sleep runs before SIGTERM, so the pod keeps listening until the rules have caught up.

containers:
- name: app
  lifecycle:
    preStop:
      sleep:
        seconds: 5

That's the native sleep action, so the image doesn't need a shell or a sleep binary.

Without a readiness probe there's a second problem on top. kubectl rollout status returned after 1.3 seconds, before any new pod was listening, and Kubernetes had already started removing old ones. That's where most of the naive setup's refused connections came from.

The failures that came after the rollout

The column on the right was the surprise. To see how long the late failures went on, I ran the readiness setup once more with a client that kept going for 170 seconds:

That's 70 timeouts in 28,505 requests. The rollout finished after about 9 seconds, so roughly two thirds of them came after it was over. Then they stopped, at about 120 seconds.

120 seconds is the node's nf_conntrack_tcp_timeout_syn_sent. Thirty seconds after one rollout I listed the conntrack table on the client's node and found entries in state SYN_SENT whose NAT still pointed at a pod that had been deleted. My reading is that each connection sent into the race window leaves a conntrack entry for a dead pod IP. When the client later picks the same source port again, the new SYN matches the old entry and follows the old NAT to the dead pod. I didn't trace individual packets, so treat that as the likely mechanism, not a proven one. What I can say for certain is that the timeouts stopped when that 120-second timer would have run out, and kube-proxy didn't clear those entries in this cluster.

The same 170-second run with the preStop hook had zero failures in 32,180 requests. If no connection is ever sent to a dead pod during the rollout, nothing is left over to hit later.

Keep-alive clients never showed this, because they open very few new connections.

The setup that ignores SIGTERM

The "no handler" setup came out at zero for fresh connections, the same as preStop. That's an accident.

The app runs as PID 1 in its container. Linux doesn't apply a signal's default action to PID 1, so without a handler SIGTERM does nothing. The pod keeps serving until the 30-second grace period runs out and the kubelet kills it with SIGKILL. For fresh connections that works like a 30-second preStop sleep. For keep-alive clients it doesn't: all 10 resets in that row, in both runs, came 30 to 40 seconds after the rollout started, when the kill cut the open connections off.

kubectl rollout status returned after 9.2 to 9.4 seconds for this setup, the same as the others. When I watched the pods during one of these rollouts, the last old pod was killed 39 seconds after the rollout started, with status Failed.

What I got wrong

The October post said a client that opens a fresh connection per request "will hide most of this". On this setup it hid nothing. It just failed differently, and for longer. That post's clients went through a cloud load balancer, so the comparison isn't exact, but the sentence was a guess and I wrote it as a fact.

When the late timeouts first showed up, I suspected DNS, because the client was resolving the Service name for every connection. I switched it to the ClusterIP and the late timeouts stayed. DNS had nothing to do with it.

I planned to show that ignoring SIGTERM makes rollouts slow, using the time kubectl rollout status takes. It took 9.4 seconds, the same as the other setups with a readiness probe. It reports when the new pods are ready, not when the old ones are gone. I had to watch the pods to see the 30 seconds.

Caveats

This is kind on one laptop, with kube-proxy in iptables mode. On a real multi-node network the endpoint removal takes longer to reach every node, so I'd expect the race window to be wider, not narrower. Clusters that use a different proxy, such as IPVS, nftables or a Cilium kube-proxy replacement, may handle stale conntrack entries differently, and I haven't tested any of them. kind made it free to run dozens of rollouts in an afternoon. The obvious next step is to repeat the 170-second run on the DigitalOcean cluster from October, where the client comes in through a load balancer.

The keep-alive runs sent fewer requests than the new-connection runs, and I didn't work out why. Compare failure counts within a table, not across tables. The client had a 3-second timeout, so each timeout here cost someone 3 seconds. A client with a longer timeout would have waited longer.

What I would take from it

• Add a short preStop sleep to anything behind a Service. Five seconds was enough here. Leave room for it in terminationGracePeriodSeconds.
• Drain open connections too. On SIGTERM, send Connection: close and finish the requests in flight before exiting. preStop handles new connections, and the drain handles the ones already open. You need both, because you don't control what kind of client you get.
• Don't stop watching error rates when the rollout finishes. Here failures went on for two minutes after it.
• If your app runs as PID 1, check that it handles SIGTERM. If it doesn't, every rollout waits out the full grace period and then cuts off connections.
• kubectl rollout status finishing doesn't mean the old pods are gone.