In June we opened a new depot, and release 7.3 of our routing service was meant to start sending it orders. The code that knew about the depot went out at eleven. The postcode mapping that pointed districts at it did not, and for three hours those orders went to the depot that had served them the day before, a hundred and twenty miles further away.
The mapping lives in a ConfigMap called depot-routing, generated from a spreadsheet the network planning team keeps. Our deploy job ran kubectl apply against the release's manifest directory, and that morning it failed with The ConfigMap depot-routing is invalid: metadata.annotations: Too long: must have at most 262144 bytes.
A ConfigMap can hold up to a megabyte. Client side kubectl apply, though, stores a complete copy of whatever it applied in an annotation, kubectl.kubernetes.io/last-applied-configuration, so that next time it can work out which fields you removed. All of an object's annotations together are limited to 256 kilobytes. The week before the mapping had been 241 kilobytes and the copy fitted. The new depot took it to 270.
The job went red, but the ConfigMap was the only object that failed. kubectl apply carries on through the rest of the files when one object is rejected and reports the errors at the end, so the Deployment with the new image had been applied a moment later. The person rerunning the job got the same error, twice, while the new code read the old mapping.
We applied the ConfigMap with kubectl apply --server-side, which tracks ownership of fields in managedFields on the server and keeps no copy in an annotation, and the routing was correct within a minute. Every deploy now uses server side apply. Configuration is applied as its own step, before any workload, and the job stops if it fails. The mapping itself has moved out of the ConfigMap into a versioned file in object storage that the service loads at startup, with the version pinned in the release. And CI fails any manifest larger than two hundred kilobytes, long before a cluster has a chance to refuse it.
Client side apply keeps a second copy of everything you give it, and the copy has the smaller limit. Ours held for two years of growth and then stopped one depot short.
– Sergey Shinder
