We added a node pool of cheaper instances on a Tuesday. A third less per hour for the same memory and roughly the same throughput, joined to the cluster in the afternoon without any ceremony because adding nodes is not a change anyone treats as interesting.
By the evening four services were in crash loops and two more were fine, and the difference between them made no sense to anybody for about an hour.
The failing pods were all on the new nodes. The message in their logs was exec format error, which is the kernel saying it has been handed a binary it cannot execute. The new instances are arm64. Our images were amd64 and had been since we built the first one.
What I had not understood properly until that evening is what a tag actually refers to. An image reference does not point at an image. It points at a manifest list, which is a small document mapping architectures to images, and the runtime on the node picks the entry that matches itself. The two services that kept working were owned by a team that had moved to a multi architecture build months earlier for their own reasons, so their manifest had two entries. Ours had one, and every node we had ever run happened to match it, so the question had never come up.
The scheduler is not at fault here and neither was anyone who signed the change off. Nothing in a pod spec says which architectures its image supports, so there is nothing for the scheduler to honour. It placed pods on nodes with room, which is its entire job.
We tainted the new pool the same night and drained it, which returned everything to normal. Then the real work. Every existing workload got a node affinity on the architecture label, so a pod cannot land somewhere its image cannot run. The services we wanted on the cheaper pool now build for both architectures in CI. And our deploy step inspects the manifest of the image it is about to ship and refuses if the architectures it advertises do not cover the pools the workload is allowed to be scheduled onto.
The cluster was heterogeneous the moment we added those nodes. Our images had been making a promise about hardware that none of us had ever read.
– Sergey Shinder
