A test named
test_all_adapters_importableasserted nothing. It would pass forever, even if every adapter was broken.
57 of 65 assertion files were in the wrong format, and the harness returned 0 / 0 without a whisper.
If a green suite makes you relax, this post is going to un-relax you.
Three Times a Green Suite Hid a Real Failure
1. The test gutted to pass. In planner-critic-engine, a test literally had a pass body:
def test_all_adapters_importable():
# if it imported, it's fine — except this proves nothing
pass
It was caught in code review, not by CI, and only before the LLM sweep because a human read it. Issue #236.
2. The harness that returned 0 / 0 and called it green. In the same repo, 57 of 65 assertion files were in the wrong format. The harness parsed them, found zero assertions to run, and returned 0 / 0 — which it treated as success. The suite was green because it had executed nothing.
3. The auth guard that never ran. In CauterRule v0.3.0 the suite reported 1,558 tests green. The MCP HTTP bearer-auth guard never ran, because an import was swallowed. Every unit test passed. The only reason we found the unauthenticated store read was a Docker field test running the real thing in a real container.
The Dangerous Thing Wasn't the Bug
In all three cases the bug mattered, but it wasn't the scariest part. The scariest part was the confidence.
We treat "tests pass" as evidence. Sometimes it's evidence that the harness silently skipped a module, or that an import was swallowed, or that a fixture file was parsed into an empty set. The suite stays green and the system ships with an unauthenticated read path.
The Fix Is a Meta-Test, Not More Tests
The fix is not to write more tests. It's to assert that your tests actually ran and actually asserted.
def test_suite_is_not_empty():
results = run_assertion_files("tests/assertions/")
assert results.executed > 0, "harness ran zero assertions"
assert results.assertions > 0, "assertions parsed to empty set"
Two rules fall out of this:
-
CI must fail when a module produces zero results.
0 / 0is an error state, not a pass. - Distinguish "did not run" from "ran and passed." A swallowed import and a skipped module should be loud failures, not silence.
That single distinction is the difference between a flaky gate and a strict one.
The Honest Limitation
Meta-tests add process, and process can rot — a meta-test that stops checking is just another green checkmark. And no amount of test discipline catches the false negatives you never thought to test for. This reduces the class of "green but broken." It does not eliminate it.
But it does close the worst category: the test that never ran and told you everything was fine.
What's the last green build you caught lying to you? I now trust a green suite about as far as I can read its raw output.
Repos and receipts: CauterRule · CauterRule v0.3.0 field test report · PlannerCritic failure-mode register · agent-tooltrust design decisions — all MIT, all public.
