In March one of our suppliers changed the date format on their delivery notes, and the Lambda that parses them started rejecting every one. We fixed the parser that afternoon. The pull request was reviewed, CI built the package, terraform apply ran green, and the next morning the Lambda was still rejecting the notes. Its logs showed a stack trace from a line we had deleted.
The function is an aws_lambda_function whose filename points at build/delivery-parser.zip. Until February the zip name carried the build number, so every build changed the filename attribute and Terraform updated the function code. A tidy up gave the zip a fixed name, and from then on apply reported No changes. Your infrastructure matches the configuration.
It was right. Terraform compares the configuration with the state and with the real resource, and the configuration was the same string as before. The provider only knows that the contents of the package have changed if you tell it, through source_code_hash, the base64 encoded SHA-256 of the zip, which it compares with the CodeSha256 that Lambda reports for the deployed code. We had never set it, because the changing filename had been doing that job without anybody knowing. Two earlier releases had gone nowhere either. They were small enough that nobody checked.
A green apply that says nothing changed reads as reassurance. On a deploy it should have read as an alarm.
We set source_code_hash to filebase64sha256 of the package, and the next apply updated the function. Then we reprocessed two days of delivery notes from the bucket. The zip is now built reproducibly, with fixed file timestamps and ordering, so its hash changes when the code changes and not on every build. After apply, the pipeline reads CodeSha256 from get-function-configuration and fails unless it matches the hash of the package it has just built. And every function publishes a version whose description carries the commit, so anyone can ask Lambda what it is actually running.
Terraform deploys what is in the configuration. The code inside our zip was not in it, until we put its hash there.
– Sergey Shinder
