We keep a public changelog, and the rule I set for it is that it has to include the things we broke. Otherwise it is marketing with a date column on it.
Here is the entry from last week I would rather not have written, and the boring thing that actually caught it.
The bug
We sell a handful of small packs. Two of them are priced the same, $149 each.
Every product on the site carries JSON-LD in the page head, and each one has an Offer block with a url that goes straight to a Stripe checkout. That is the path a lot of machine traffic takes now. Search engines read it, and so do the AI assistants that answer "where do I buy X" without anybody clicking a link.
Those two url values were swapped. Pack A's structured data pointed at Pack B's checkout, and the other way around.
Why nothing looked wrong
This is the part worth sitting with. Both packs cost $149.
So the checkout opened, and it said $149, which is the number the buyer expected. The Stripe page was real. The payment would have gone through. Nothing in the flow contradicted anything the buyer had read. They would have paid the correct price and been handed the wrong product, and the first person to find out would have been them, after paying.
If the two packs had been priced differently the mismatch would have been obvious in about four seconds. Matching prices is what made it invisible.
I have no evidence anyone hit it. I also have no evidence nobody did, which is a worse sentence to write than the first one.
What found it
Not a test. Not a customer email. A script that does one dumb thing.
It fetches the live page, parses every JSON-LD block, collects every Offer.url, and asserts that the count of offers equals the count of distinct URLs. That is the whole check.
offers: 12 distinct urls: 12
When those two numbers disagree, two products are pointing at the same checkout, or at each other's. It cannot tell you which, and it does not need to. It tells you to go look.
It runs against production, not a fixture, because the thing I care about is what the page is actually serving right now. A fixture would have passed happily the entire time the live site was wrong.
The general shape
The checks that have earned their keep here all have the same feel. They are not "does this function return the right value." They are closer to "do two things that must agree still agree."
A few that now run before every deploy:
- Every
Offer.urlis distinct. - Every page's
dateModifiedequals itslastmodin the sitemap. Two freshness signals that drift apart are worse than one. - Every internal link on a page returns 200. Cheap, and it catches renames.
- Every secret the build needs was actually bound at the last deploy, read back from the platform rather than from the config file that claims to set it.
None of these are clever. Every one of them has caught something real, and each one exists because it caught something the first time I ran it.
The crossed checkout is the one I think about, though, because it is the only one where a paying customer was the next line of defense. The other bugs on that list were embarrassing. That one was going to cost somebody money and take their afternoon.
The uncomfortable part
I found this while writing a changelog entry about something else.
The honest version of ship-in-public is that the changelog itself is a check. Writing down what shipped forces you to go look at whether it shipped the way you think it did, and about one time in five, it did not.
Full entry and the rest of the log: https://kit.sdvsignal.com/changelog/
