I fixed a bug where two places in a system computed the same selection logic, and one of them did not get updated when the other changed. I added a regression test afterward: 12 test cases, all passing. I sent it for review.
The review came back with one line: the test that was supposed to prove the fix had reimplemented the selection logic instead of calling the real function that does the selecting.
What I actually wrote
I wrote a small loop inside the test file that mirrored what I believed the real selection function did: same conditions, same order, same output shape. Then I asserted the dry-run result matched that loop's result.
That passed. All 12 cases passed. It looked like coverage.
Why that does not prove anything
The bug I had just fixed was exactly this shape: the same decision made in two places, one changed, the other did not follow. Writing a second copy of the decision inside the test file recreates the identical risk one level up. The test was not checking whether the real function behaves correctly — it was checking whether my copy of the function agrees with my other copy of the function. Both copies could be wrong in the same way and the test would still pass.
The fix
I wrote a separate test that calls the real function directly — no duplicated logic, no simplified stand-in. The only things replaced were the parts with side effects: the function that actually sends output, a network lookup, and file writes. Those were swapped for stubs that record what they were called with and do nothing else. The selection logic itself runs unmodified, exactly as it runs in the real implementation.
What I actually verified
I ran a reverse probe: removed the one line in the implementation that was the actual fix, then ran both test files against that broken code.
old test file (rewritten logic): 12 passed
new test file (calls the real function): 1 failed, 3 passed
The old file passed 12 out of 12 with the bug back in place. The new file caught it: 1 failed, 3 passed. That is the difference between a test that checks the code and a test that checks a copy of itself.
What this does not prove
This confirms the new test can detect this one specific regression. It does not prove the new test catches every way this function could break, and it does not prove the old test is useless everywhere — only that it gave a false pass on exactly the failure it was written to catch. A test suite can hold both kinds of test at once, and the pass count alone will not tell you which kind you are looking at.
What I check now
Before trusting a passed result on any test meant to verify logic, I look at the test file for an if, for, or selection loop that mirrors the intent of the real implementation, instead of a direct call to the real function followed by an assertion on its output. That shape is the signal.
The fastest way to check is the same reverse probe: remove the line in the implementation that matters, rerun everything, and see what turns red. If nothing does, the test's pass rate was never measuring the thing it claimed to measure. It was measuring agreement between two versions of the same guess.
This article was written with the help of AI. The incident, the commands and the outputs are real and from my own machine; the draft was AI-written, then checked against the original logs and corrected.
