At nine on a Monday in March, every one of our trade partners' systems started getting 400 Bad Request from our order intake API, with the body missing partner id. Over the weekend we had moved the API from an Amazon API Gateway REST API to an HTTP API, which costs less and suited what the API does.
Partners identify themselves with an X-Partner-Id header. Our Lambda read it with event["headers"]["X-Partner-Id"], and if the key was not there it returned that 400. With the REST API the header names arrived as the partner sent them. The new integration uses payload format 2.0, and its documentation says plainly that header names are lowercased. Our handler looked for X-Partner-Id in a dictionary that now held x-partner-id, and treated every partner as anonymous.
Nothing in HTTP made our code correct before. Header names have always been case insensitive, RFC 9110 says so, and HTTP/2 sends them lowercase on the wire anyway. The REST API had preserved the case, our code had come to depend on it, and a Python dictionary compares keys exactly.
Our tests passed because their event fixtures were typed by hand, with the header in the capitalisation we expected. The staging smoke test passed because it called the health endpoint, which needs no partner. We rolled back the DNS at twenty past nine and had orders flowing again by half past, and the partners who retried automatically filled our queue for the rest of the morning.
The handler now normalises header names to lowercase once, at the edge, and reads only the normalised copy. Test events are captured from the real gateway in staging, not written by hand, and the suite runs each one through every payload format we could be given. The smoke test after any gateway change places a real order for a test partner and cancels it. And our client libraries and examples send the header in three different capitalisations on purpose, so nothing on our side can grow fond of one.
The header was the same header in both gateways. What changed was the spelling, and we had built our code on the spelling.
– Sergey Shinder
