Last february we swapped our embedding model from text-embedding-3-small to bge-large, thinking "better embedder, better RAG." our golden set stayed green. two negative tests that had been failing for months suddenly passed. looked like a win.
When the retriever fails to pull the relevant chunks, the LLM never sees the trap. if it's a polite model, it refuses to answer, and your test passes. but it passed for the wrong reason: the retrieval failed, not the model. the test looked green while the system was
broken.
This is the same shape as Serhiy Kucherenko's finding that "the corpus decides more than the model does," just quieter. his refusal traps worked because the prompt boundary held. ours failed silently because the retrieval boundary broke.
The fix: trap_chunk_id and embedder tags
Every negative test in our golden set now carries two stamps:
- trap_chunk_id: the id of the chunk containing the trap, stamped at authoring time. our CI step fails the run if that chunk isn't in the retrieved set, regardless of what the LLM says. "test did not run" becomes a third verdict next to pass and fail.
- embedder tag: the name of the embedding model under which the test was last validated. after any swap, lines with the old tag are stale by default — they stay in the file, but the CI summary flags them as "unverified under current embedder" until someone re-runs and re-dates them.
The mechanics are dumb simple:
# golden_set.json
{
"question_id": "neg_17",
"question": "what was the q3 2019 revenue?",
"expected_answer": "not in documents",
"trap_chunk_id": "chunk_847",
"embedder_tag": "text-embedding-3-small-2024-02-15",
"last_validated": "2024-02-15"
}
# ci_eval.py
retrieved_chunk_ids = run_retrieval(question)
if trap_chunk_id not in retrieved_chunk_ids:
verdict = "test did not run"
elif llm_answer == expected_answer:
verdict = "pass"
else:
verdict = "fail"
When we swapped to bge-large in february, the two negative tests that "passed" immediately turned red: trap_chunk_id wasn't in the retrieved set. the new embedder was worse at semantic similarity for those specific traps. we rolled back the swap, fixed the chunking strategy, and only then re-validated the tests under the new embedder.
The maintenance cost nobody talks about
The catch is real: trap_chunk_id moves whenever you re-chunk or re-parse documents. every pipeline change triggers a re-stamping step:
- re-run the full document ingestion
- for each negative test, find the new chunk id containing the trap
- update the golden set with the new trap_chunk_id
- re-run eval under the current embedder
- update the embedder_tag and last_validated timestamp
It's maybe 20 minutes of work per pipeline change, but it's non-negotiable. skip it and you're trusting verdicts that describe a retrieval that no longer exists.
Same goes for the embedder tag: after any swap, every line with the old tag is stale. our CI summary now has a section called "unverified under current embedder" that lists all the tests waiting for re-validation. cheap bookkeeping, but it stopped us from shipping broken retrieval twice.
Why cosine similarity can't save you
Amritpal singh wrote about this exact problem: his refusal threshold had no effect because the similarity distributions for answerable and unanswerable questions overlapped. cosine similarity can't tell "about the topic" from "answers the question." both get high scores.
Our trap_chunk_id approach sidesteps the overlap problem entirely. instead of trying to find a threshold that separates the distributions (which doesn't exist), we assert that the specific chunk containing the trap must be retrieved. if it's not, the test didn't run.
This is the same insight as hybrid search: single-vector similarity will always fail on negatives because high similarity ≠ correct answer. BM25 catches exact matches, vectors catch semantic similarity, metadata filters catch structural context. but for negative tests, none of that matters if the trap chunk isn't retrieved at all.
The checklist
If you're running negative tests in your RAG eval:
- every negative test carries a trap_chunk_id
- CI fails if that chunk isn't retrieved (verdict: "test did not run")
- every test carries an embedder_tag from when it was last validated
- after any embedder swap, tests with the old tag are flagged as stale
- re-chunking triggers a re-stamping of all trap_chunk_ids
- the CI summary shows "unverified under current embedder" tests prominently
If your eval doesn't have these, you're probably shipping retrieval bugs that look like model bugs.
The broader lesson
Edward Izgorodin's framing is the sharpest I've seen: "writing the correction down is not the fix, because the memory that carried the mistake is still in the store and still eligible." But there's a subtler failure mode: evaluation fails without observability into whether the test actually ran.
A negative test that passes because the LLM refused is not evidence that the retrieval worked. it's evidence that the LLM is polite. the only way to know if the retrieval worked is to check whether the trap chunk was actually retrieved.
This is the same shape as observability in production: you don't trust that a service is healthy because it returns 200 OK. you check that it actually did the work. same logic applies to eval.
Where to go deeper
This post is the mechanics. if you want the full audit checklist for production RAG (parsing, chunking, metadata, hybrid search, reranking, eval), i packaged it as a PDF: "RAG checklist: 10 checks to stop LLM hallucinations on tables and numbers" plus a golden set template with 20 questions covering digits, contracts, dates, comparisons, negatives, and paraphrases.
Both are free in my telegram channel: @llmops_engineering — comment "+" on the pinned post and i'll send you both PDFs. no email signup, no spam, just the files.
The channel is where i post production RAG war stories, vector DB benchmarks, and the boring LLMOps stuff that actually ships. if this post saved you a week of debugging, the channel will save you a month.
What's your eval setup for negative tests? have you hit the "test passed but retrieval was broken" failure mode? drop your approach in the comments — curious how other teams handle this.
