I wrote a script whose entire job is to break my system on purpose in a sandbox and print what every surface says. scripts/broken-lab.sh, 516 lines, committed as ca66f6c8. It found three real defects in its first hour, which felt great until I looked closer and found that three of the bugs it found were its own.
It wrote three rows into the live gateway.db. It health-checked a different process than the one it started. And it leaked eight daemons into a machine-wide process limit that the production stack shares.
The system under test was fine. The instrument was lying.
The harness wrote to production because the app resolved its own path
The gateway reads VODOU_PROJECT_PATH to decide where its SQLite file lives. What I did not know is that db.ts only trusts that variable if the directory already contains vodou-core.db, and it resolves the root once at module load. My lab set the env var, skipped the baseline step that seeds that file, and the gateway silently fell back to the repo root. Then it staged a graph fan and recorded three runs in the live database: the one file this script promises in its header never to touch.
Nothing errored. The harness passed. I deleted the rows by hand.
The fix is not "set the variable more carefully." The fix is that the harness stops believing it:
if port_is_taken; then
# Isolation is asserted, never assumed. If the gateway created its DB
# anywhere but the lab, every later reading describes the live system.
if [ ! -f "$LAB/MCP-servers/Vodou-Console/gateway.db" ]; then
say "ISOLATION FAILED — the lab gateway did not create $LAB/MCP-servers/Vodou-Console/gateway.db."
say " It is writing to the LIVE databases. Stopping before anything else runs."
lab_gateway_kill
return 1
fi
return 0
fi
A 200 on your port is not proof that it's your process
Second bug, and this one cost a full debugging cycle. Startup waited for /api/health to return 200 and called that success. But a leftover gateway from an earlier lab run still held :8791, so my new process logged Refusing to start a second instance, exited, and the health probe cheerfully passed: against the stranger. Every subsequent request hit another lab's database. The new lab's gateway.db was never created. Then the kill test SIGKILLed a pid that had already died and reported a clean recovery.
Now the loop checks kill -0 "$pid" on the pid it spawned before it believes any 200, and aborts outright if the port is taken before boot.
The third bug: the lab gateway spawns its daemon and worker detached, from the repo binary, so kill -9 -PGID misses them and pkill -f "$LAB/vodou-core" can never match. The lab's path isn't in their argv. Eight orphans accumulated in one afternoon, and since VODOU_MAX_PROCESSES counts machine-wide, the next run was refused with 6 vodou-core processes are already running (limit 5). The harness had started starving the machine it promises not to touch. The only place the lab's identity survives is the environment, so that's what the reaper matches now: ps -E -o pid=,command= | grep -F "VODOU_PROJECT_PATH=$LAB".
Any harness that discovers its dependencies like production will grade production
The class: any harness that discovers its dependencies the same way production does will eventually grade production. Not "uses the same database". discovers the same way. Config file, env var with a fallback, service discovery, a well-known port, a shared process pool. It shows up as pytest fixtures against the shared dev Postgres, LangChain eval suites pointed at the production vector store by an unset INDEX_NAME, Compose suites reusing host ports, and any MCP or daemon test that spawns subprocesses with no cleanup barrier.
Testcontainers: Your Isolated Tests Are Still Lying To You is right that a shared dev database is a solved problem and half-isolation is the real trap. What it misses is my case: a throwaway container guarantees a clean database exists, not that the app under test used it. My isolation was correct in the harness and discarded by the callee. Closer is Your test suite is lying to you about which process it's in, where a correct guard ran in the wrong process. Same family. My version is worse in one way: I didn't have a guard that ran in the wrong place, I had a probe that couldn't tell processes apart at all.
A harness must assert on every run that the process answering it is the pid it spawned, and that the writes it made landed under the path it created. Both are resolved by the system under test, not by the harness. That's checkable: grep your test setup for a readiness probe with no pid comparison, and for a teardown that trusts argv.
Five-minute check against your own stack, no Vodou anything:
# 1. Does your suite touch production? Fingerprint before and after.
psql "$PROD_URL" -tAc \
"select relname, n_live_tup from pg_stat_user_tables order by 1" > /tmp/before
pytest -q # or npm test, go test ./...
psql "$PROD_URL" -tAc \
"select relname, n_live_tup from pg_stat_user_tables order by 1" > /tmp/after
diff /tmp/before /tmp/after && echo ISOLATED || echo "YOUR SUITE WRITES TO PROD"
# 2. Is the thing answering your health check the thing you started?
./run-test-server & echo $! > /tmp/mine.pid
curl -sf localhost:8080/health >/dev/null && echo "health says 200"
lsof -ti tcp:8080 -sTCP:LISTEN > /tmp/listening
grep -qx "$(cat /tmp/mine.pid)" /tmp/listening \
&& echo "same process" || echo "TESTING A STRANGER: $(cat /tmp/listening)"
# 3. Do you leak? Count before and after.
pgrep -fc myservice # run this before the suite, and again after
Passing looks like: empty diff, "same process", identical counts. Failing looks like a table that grew by three rows nobody can explain, a pid in /tmp/listening that isn't yours, and a count that climbs one per run until something unrelated starts getting refused. (n_live_tup is an estimate; for small tables use exact count(*) per table.)
The rule I'd give a stranger: a test that cannot prove which process it talked to and which file it wrote has not produced a result, it has produced a sentence. Make isolation an assertion with an abort, not a configuration you set once and trust.
Source: Your isolated test lab isn't isolated: 3 rows, 8 daemons by Chad Priest, from Building Vodou in Public.
