In June every nightly database export failed with AccessDenied on kms:GenerateDataKey. The role that ran them had not changed. Its name, its permissions and its trust policy were exactly what they had been the night before.
That week we had moved our CI role from an old bootstrap Terraform state into the platform state. We did it the quick way: deleted the block from one configuration, applied, and created the role from the other a few minutes later, with the same name and the same policies. To IAM it was a different role.
Every IAM role has a unique ID, beginning with AROA, as well as its name. When you save a resource based policy, a KMS key policy or an S3 bucket policy, that names a role by ARN, AWS stores the role's unique ID behind it. If the role is deleted, the policy shows the bare ID, and a new role with the same name has a new ID that the policy does not match. Our backup keys had key policies listing the CI role and nothing else. Years earlier somebody had removed the default statement that grants the account root access, because it looked too broad. Without it, IAM permissions count for nothing on that key, and the only principal the policy trusted no longer existed. Nobody in our account could change the policy, including administrators. Getting it back took a support case with AWS and three days without encrypted exports.
Every key policy we own now keeps the statement for the account root, so a key can always be managed through IAM, and administration goes to a break glass role that is never deleted, not to a pipeline. Roles move between states with removed and import blocks, never by delete and create. A check reads the plan and fails any destroy or replace of an IAM role that any key or bucket policy refers to. And a weekly job reads every key and bucket policy and alerts on a principal shown as a bare unique ID, which is what a policy looks like when the role it trusted has gone.
A policy that names a role trusts one particular role, not whatever is called by that name later. Ours trusted a role we had deleted, and only AWS could tell it otherwise.
– Sergey Shinder
