I spent an evening convinced GitHub was sending broken webhook signatures. It wasn't. echo was.
The setup
Chngd's GitHub Marketplace webhook verifies every inbound event with HMAC-SHA256, comparing GitHub's X-Hub-Signature-256 header against a signature computed from the raw request body and a shared secret.
Every single request failed verification. Not intermittently. Every one, from the very first test delivery.
Where I looked first
The obvious suspects, in order:
- Was I hashing the raw body, or a re-serialized/re-parsed version? Framework body-parsers are notorious for this: re-serializing JSON changes byte-for-byte content even when the data is "the same." Checked: raw body, untouched.
- Wrong hashing algorithm or encoding (hex vs base64)? Checked against GitHub's docs: correct.
- Timing-safe comparison bug? Checked: standard constant-time compare, no typo.
All correct. And it still failed, every time, deterministically. That determinism should have been the tell. A subtle logic bug fails sometimes or on specific payloads, not literally always.
Where it actually was
The secret itself was wrong. Not the value. The bytes.
I'd generated it and piped it into the environment file with echo $SECRET >> .env. echo appends a trailing newline by default. That newline became part of the stored secret string. GitHub's config field has no trailing newline, so their computed signature was based on the clean secret. My server was hashing with the secret plus an invisible \n, so every signature it computed was simply wrong, no matter what payload came through.
Nothing in any error message hinted at this. A signature mismatch looks identical whether the payload is wrong, the algorithm is wrong, or the secret has one invisible extra byte at the end.
The fix
printf '%s' "$SECRET" >> .env instead of echo. printf doesn't add anything you didn't put there.
The actual debugging move
If you hit a signature mismatch, don't start by re-deriving your verification code. Print the secret's length and a repr/quoted-string dump of it server-side first. A trailing newline shows up immediately in a repr; it's invisible everywhere else. That one check would have saved most of the evening.
The structural problem
echo vs printf is the proximate cause, not the real one. The real one is that a secret transited a human shell session and a hand-edited file with no validation step. Two things follow from that:
- Secrets shouldn't be typed or piped through a terminal where avoidable. Generate them, inject via a secrets manager or CI/CD secret store, and verify the stored value (byte length, or a known-good HMAC against a test vector) before it ships.
- Appending to a .env file from a shell is fragile on its own terms, independent of this bug. If the variable already exists in the file, you now have two entries, and which one wins depends on parser load order, not intent.
Neither of those is exotic. Both would have caught this before it ever reached "why is GitHub sending bad signatures."
