⚡ TL;DR: An EKS worker node running 30+ pods crashed at 08:43 IST on Aug 16. EC2 Auto Scaling didn't act. EKS Node Auto Repair was never enabled. Cluster Autoscaler had no headroom (Min=Max=6). Our memory alert fired 82 seconds after the node was already dead. We fixed it manually at 12:30 IST — 4 hours later. Every fix is one or two CLI commands. Here's the full breakdown.
When an EKS worker node goes down, Kubernetes is supposed to handle it. That's the whole pitch — self-healing infrastructure. Pods evict, workloads reschedule, the cluster heals itself.
In a recent incident, a node running 30+ pods crashed at 08:43 IST. None of the four auto-recovery mechanisms we had in place did anything. The node sat NotReady for almost 4 hours. We discovered it 82 seconds after it died — from a memory alert that fired when the node was already gone — and fixed it manually at 12:30 IST.
The interesting part wasn't that the node crashed. It was that four different layers of protection failed for four completely different reasons.
Here's exactly what failed and why.
The Node That Looked Healthy Until It Didn't
ip-10---*.ec2.internal had been running since Jun 28. From Kubernetes' perspective, it was Ready. CPU requests at ~97%. Memory requests at ~69%. Nothing alarming.
The actual picture was very different:
Allocated resources:
Resource Requests Limits
cpu ~97% ~545%
memory ~69% ~358%
358% memory limit overcommit. Kubernetes scheduled 37 pods there because requests showed 69% — but if those pods pushed toward their limits, the node had no chance. The requests were lies; the limits were the truth. And the truth said this node was 3.5x overloaded.
Underneath the Ready status, the containerd runtime had been in a degraded state since Jul 20, when the OOM killer had hard-killed a MongoDB process. The kernel had logged two containerd-shim deadlocks (Jul 1 and Jul 29) that the node had somehow absorbed. Neither showed up anywhere in monitoring. The control plane saw nothing wrong.
On Aug 16 at 08:22 IST, a service responsible for processing audit events started leaking memory. ELK showed node memory at 81.6% at 08:22:57 IST. By 08:43 IST the service had grown from 2.0GB to 6.2GB — a 4.2GB spike in 21 minutes — pushing the node from ~81% to ~96%.
The memory pressure triggered a third containerd-shim deadlock on the already-corrupted runtime. This time it was fatal. At 08:43:22 IST, SSM lost contact. Kubelet posted its last heartbeat. At 08:45:10, the node went NotReady.
Now watch how every auto-recovery mechanism missed it.
Failure #1 — EC2 Auto Scaling Health Check
The ASG protecting this nodegroup was configured with EC2-level health checks:
HealthCheckType: EC2
Min: 6 | Max: 6 | Desired: 6
EC2 health checks have exactly one question: is the VM powered on?
aws ec2 describe-instance-status \
--instance-ids <id> --region <region> \
--query 'InstanceStatuses[].[InstanceState.Name,SystemStatus.Status,InstanceStatus.Status]' \
--output table
+---------+------+------+
| running| ok | ok |
+---------+------+------+
The instance was running. System status: ok. Instance status: ok. The AWS hypervisor was satisfied. The deadlock inside containerd was completely invisible at the hypervisor layer — the kernel was alive, the VM was up, the NIC was responding to health probes.
EC2 health check never triggered. No replacement instance was launched.
The lesson: EC2 health checks only detect hardware failure or VM termination. They cannot detect OS-level hangs, kubelet death, containerd deadlocks, or any software failure that doesn't take down the underlying VM. For Kubernetes node health, EC2 checks are nearly useless.
Failure #2 — EKS Node Auto Repair
EKS Node Auto Repair is designed exactly for this scenario. When a node stays NotReady for a defined period, it automatically cordons, drains, and replaces it. No manual intervention needed.
aws eks describe-nodegroup \
--cluster-name <name> \
--nodegroup-name <nodegroup_name> \
--query 'nodegroup.nodeRepairConfig'
null
null. Never configured. Never enabled.
This feature was available. We had just never set it up. The node sat NotReady for nearly 4 hours while the feature that would have replaced it in 10 minutes was turned off by default.
The lesson: EKS Node Auto Repair is not enabled by default. Check every nodegroup right now with the command above. If you get null, you have this gap.
Failure #3 — Cluster Autoscaler
The cluster autoscaler was running. It's supposed to handle exactly this kind of situation. But our configuration made it powerless:
Min: 6 | Max: 6 | Desired: 6
Min equals Max. The autoscaler has no headroom to operate. To replace a broken node, it needs to launch a new one first (going to 7, violating Max), drain the broken node, then terminate it (back to 6). With Max=6, it couldn't even start that sequence.
The autoscaler logs confirmed it saw the problem — and couldn't act:
I0817 12:17:57 pre_filtering_processor.go:67]
Skipping ip-*-*-*-*.ec2.internal — node group min size reached
(current: 6, min: 6)
It knew the node was bad. It had no authority to fix it.
There was a second problem: the cluster-autoscaler pod itself was running on the broken node, stuck in Terminating state alongside 25+ other pods.
kube-system cluster-autoscaler-5********8 1/1 Terminating 0 6d5h
The thing responsible for replacing the broken node was stuck on the broken node.
The lesson: Min=Max is a zero-headroom configuration that paralyzes automated recovery. Set Max to at least Min+2 on every nodegroup.
Failure #4 — Memory Alert (Fired Too Late to Matter)
We had a memory alert configured — running on an 11-minute cron interval, watching for node memory usage above 95%.
ELK showed the node at 81.6% for hours before the incident:
IST timestamp Node memory %
08:22:57 81.6%
08:33:57 81.6%
08:44:57 80.6% ← node already down, stale value
The 4.2GB memory spike happened entirely between two check cycles. At 08:22 the node was fine. At 08:44 the check returned stale data because the node was already gone.
The alert fired at 08:44 IST. The node went unreachable at 08:43:22 IST. The alert was 82 seconds behind.
This is a structural problem: the metric source is the node itself. When the node dies, metrics stop. If the crash happens between alert evaluation cycles, you miss it entirely. The faster a node crashes, the more likely it dies between evaluations.
The lesson: Metric-based node alerts have a fundamental blind spot. If the node is the source of the metric, the metric disappears when the node dies.
What We're Fixing
- Enable EKS Node Auto Repair (do this first)
aws eks update-nodegroup-config \
--cluster-name <name> \
--nodegroup-name <nodegroup_name> \
--node-repair-config enabled=true \
--region us-east-1
When a node stays NotReady for the configured threshold, EKS automatically cordons, drains, and replaces it. The control plane makes this decision — it doesn't depend on the broken node reporting its own failure.
- Set Max > Min on Every ASG
aws autoscaling update-auto-scaling-group \
--auto-scaling-group-name <name> \
--min-size 6 \
--max-size 8 \
--region us-east-1
Max needs to be at least Min+2 to give the autoscaler room to launch a replacement before removing the broken instance.
- Switch to ELB Health Checks
aws autoscaling update-auto-scaling-group \
--auto-scaling-group-name <name> \
--health-check-type ELB \
--health-check-grace-period 300 \
--region us-east-1
ELB health checks evaluate actual application-layer response, not just hypervisor state.
- Replace Metric-Based Node Alert with Condition-Based Alert
# This fires even when the node is completely dead.
# kube-state-metrics reads from the API server, not from the node itself.
- alert: NodeNotReady
expr: kube_node_status_condition{condition="Ready",status="true"} == 0
for: 90s
labels:
severity: critical
annotations:
summary: "Node {{ $labels.node }} NotReady for >90s — EKS Auto Repair should engage"
kube-state-metrics watches the Kubernetes API server — not the node. It fires even when the node is completely unreachable.
- Enforce Memory Overcommit Limits
yaml
apiVersion: v1
kind: LimitRange
metadata:
name: memory-ratio-limit
namespace: your-namespace
spec:
limits:
- type: Container
maxLimitRequestRatio:
memory: "4" # limit cannot exceed 4x request
This forces the actual memory ceiling to stay within a predictable range of what the scheduler sees. A pod with a 1Gi request cannot have a 16Gi limit.
The Broader Problem
This wasn't an unusual incident. A node accumulated damage silently for 46 days. Nothing alerted. Nothing flagged the degraded containerd state. The node reported Ready to the control plane while internally the runtime was already compromised.
When the fatal event came — a memory spike from an unrelated service — four recovery mechanisms failed in different ways:
- EC2 health checks were asking the wrong question (is the VM up?)
- EKS Auto Repair was never turned on
- The autoscaler had no room to maneuver (Min=Max)
- The metric-based alert fired 82 seconds after the node was already dead
Any one of these gaps would have slowed recovery. All four together meant 4 hours of a NotReady node and a manual reboot at noon.
The fixes are each one or two commands. None of them are complicated. They just require knowing the gaps exist.
Check Your Cluster Right Now
Run this on every nodegroup:
aws eks describe-nodegroup \
--cluster-name <your-cluster> \
--nodegroup-name <your-nodegroup> \
--query 'nodegroup.nodeRepairConfig'
If you get null — you have this gap. Fix it before the next 2 AM alert.
The names, cluster/node identifiers, timestamps, instance IDs, pod counts, resource values, and other incident-specific details in this article have been intentionally changed or fictionalized for privacy and security. The failure pattern and technical lessons remain representative of the original incident.
Follow me on Medium for more incident stories from production.
![Kubernetes Networking [Level-9: Kubernetes Networking Internals]](https://media2.dev.to/dynamic/image/width=1000,height=420,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2m9l5prpnyvigc09fifh.png)
