A colleague made a small change to the networking workspace on a Wednesday afternoon. The pipeline was busy, he had the credentials, so he ran it from his machine, and it worked. His machine was three weeks old and had the current Terraform on it. Our CI image pins 1.7.5.
Nobody noticed until the next pipeline run, which refused to start: the state snapshot was created by a newer version and cannot be read. State format upgrades are one way. There is no downgrade command, and the file in the bucket was now something only a binary none of our automation had could open. One workspace, the one that owns every VPC, subnet and security group we run.
We found out it mattered on the Friday, when a partner changed the address range they call us from and the security group that lets them in lives in exactly that workspace. About forty minutes of that incident was three engineers working out which of the bad options to take. Upgrading the CI image for everything would have moved sixty other workspaces onto an untested version in the middle of an incident. Hand editing the version field inside the state is a thing people do and a thing vendors tell you not to. We took the previous object version out of the bucket, which was two applies old, then imported the two resources that had been created since and reconciled by plan.
What changed afterwards is mostly about who is allowed to write. The state backend now accepts writes only from the pipeline's role, and every human credential against it is read only, which removes the entire category rather than asking people to be careful. The version lives in a file that both the CI image build and the local tooling read, so there is one number. A preflight step compares the CLI it is about to run with the version recorded in the state and fails loudly on a mismatch. And the pin moves through a scheduled pull request that plans every workspace before anyone merges it.
Shared state means the tool version is a shared decision. Ours was being made by whoever most recently let a package manager update something.
– Sergey Shinder
