At 09:12 on a Tuesday every change to our production cluster started failing with the same message: the API server could not call our policy webhook, connection refused. Nothing that was already running was affected. Traffic was served, pods stayed up, customers noticed nothing. We simply could not change anything at all.
No deploys. No configuration updates. No manual scaling, and, more importantly, no automatic scaling either, because creating a pod is a change like any other and every one of them went through the same rejected admission call.
The webhook enforces our policy rules and is configured to fail closed, which is the correct setting for a thing whose job is to keep bad manifests out. Its two replicas had been running on two nodes that the cluster autoscaler consolidated overnight. Both pods were evicted. Their replacements could not be created, because creating a pod requires admission, and admission requires the webhook, whose pods are what we were trying to create. The cluster had locked itself out with a loop three objects long.
The way out is to delete the webhook configuration object, which is not itself subject to admission. Everything reconciled within a minute and we reinstalled the policy afterwards. Twenty three minutes end to end, most of it spent talking ourselves into deleting the component that enforces our security rules during what might, at that point, still have been an attack.
The configuration now excludes the namespace the webhook runs in and the system namespaces, so it can always replace itself. Its pods carry a system critical priority class, a disruption budget and anti affinity, so no autoscaler consolidation can take both at once. And there is a written, tested two command recovery for the next time, because we found out at 09:20 that nobody wants to invent that procedure live.
Afterwards we drew the dependency graph of our recovery path rather than our serving path, and asked of each step whether it could survive the failure it was meant to fix. Our deployment pipeline pulls images from a registry that runs in the same cluster. That one is still on the list.
A control that fails closed blocks bad changes and good ones at exactly the same rate, and it has no way of knowing which kind you need.
– Sergey Shinder
