Deploys started failing on a Thursday morning. All of them, in every namespace, with the same message from our rollout tool: timed out waiting for pods. The pods were healthy and running. The cluster simply could not answer a question about them inside thirty seconds.
Our import controller creates a Kubernetes Job for each file a customer sends us, roughly nine hundred a day since early 2024. It creates them through the API itself rather than through a CronJob, which matters more than it sounds. A CronJob keeps three successful runs by default and deletes the rest. A Job created directly keeps everything forever unless you set ttlSecondsAfterFinished, and we had not, because none of us knew it existed. Each finished Job also leaves its Pod object behind. We were carrying about three hundred and forty thousand Jobs and a similar number of Pods, none of which had done anything for months.
Every one of those is a row in etcd that the API server keeps in a watch cache and that every controller in the cluster lists on resync. Our etcd database had reached six point four gigabytes. List calls against large namespaces began paging for tens of seconds, the scheduler's resyncs took minutes, and our rollout tool, which lists pods before it watches them, gave up. Nothing was broken in a way that produced a single obvious error. Everything was slow in a way that expressed itself as unrelated timeouts.
Deleting them was its own operation, because three hundred and forty thousand deletions are three hundred and forty thousand writes into the database that is already struggling. We removed them oldest first in batches of five hundred with a pause between, over two days, then compacted and defragmented each etcd member in turn.
Now every Job the controller creates carries a one hour TTL, an admission policy rejects any Job submitted without one, and an alert fires when the object count for any resource type in a namespace passes five thousand.
A Kubernetes object is a database row that every component in the cluster reads. Anything in your system that creates objects at a rate needs a rule for removing them, written on the same day, because the platform will keep them faithfully until it cannot.
– Sergey Shinder
