In April we sold for four days at last month's prices. Our largest supplier had raised theirs on the first of the month and sent the new price files as usual. They had arrived. They had just never been loaded.
The supplier uploads over SFTP into an S3 bucket. Each new object triggers an event notification, which invokes a Lambda that reads the object key from the event, fetches the file and loads the prices. That week the supplier upgraded their system, and the files changed from PRICES_20260401.csv to Price List 2026-04-01.csv.
The object key in an S3 event notification is URL encoded, and the AWS documentation says so with exactly this example: a space arrives as a plus sign. Our Lambda took the key as it came and asked for Price+List+2026-04-01.csv, which did not exist. It did not get a 404. Its role had s3:GetObject on the bucket but not s3:ListBucket, and without list permission S3 answers a request for a missing key with 403 Access Denied, so that a caller cannot find out which keys exist. We spent the first day reading IAM policies, the bucket policy and the key policy, all of which were fine.
The rest of the delay was ours too. S3 invokes Lambda asynchronously, and a failed asynchronous invocation is retried twice and then dropped unless the function has somewhere to send failures. Ours had nowhere. The files sat in the bucket and the events that announced them were gone.
The handler now decodes the key with unquote_plus before using it, and its tests use file names with spaces, plus signs, ampersands and accented letters. The role has ListBucket on the inbound prefix, so a missing object is reported as missing. Failed invocations go to an SQS queue with an alert on its depth. And an hourly job lists the inbound prefix, compares it with the files we have loaded, and alerts on anything older than thirty minutes that we have not.
An event about a file is a description of it, written in someone else's encoding. Ours described the file correctly, and we read it as if it were the name.
– Sergey Shinder
