A change to our shared networking module moved a resource from a counted list to a keyed map. The author planned it against the workspace he works in, which showed no changes at all, two of us approved it, and it went out as a new module tag. Three weeks later a colleague bumping a different workspace to that tag read a plan that proposed to replace a NAT gateway, which is twenty minutes with no outbound connectivity for everything in that account. He stopped, which is the only reason this is a story about a plan and not about an outage.
The module has eleven callers. They pass different combinations of about fourteen variables, three of which switch whole resources on and off, and the state each of them holds was written at a different time by a different version. A module change is not one change. It is eleven changes that will happen on eleven different days, and the review we had performed covered exactly one of them, chosen because it was the one already open in somebody's terminal. Everything else we ran on the pull request, formatting, a linter, validate, checks the code as text. None of it instantiates the module against real state, so none of it can tell you what will be destroyed.
Module pull requests now run a plan against all eleven consuming workspaces, in parallel, with read only credentials, and post one table: workspace, resources to add, change, destroy and replace. Twelve minutes, and any destroy or replace anywhere fails the pull request until somebody writes down why in the description. Separately there is a job in a sandbox account that applies the module across four variable combinations, applies it a second time and asserts the plan is empty, then destroys everything. Four pounds a run, and in its first week it caught a naming expression that produced a different result on the second apply.
Shared infrastructure code has a fan out that the code in front of you does not show. Ours looked like one file with one reviewer and was in fact a change to eleven production environments, queued up to arrive whenever each of them next happened to run.
– Sergey Shinder
