One Retrieval Method Is Not a Diagnosis

Retrieval Failure Analysis, Part 2

In the last article, every chunking strategy failed the same question. A customer cancels a TechNova order after it ships because TechNova sent the wrong item, and the question is whether the 15% restocking fee applies. Answering it takes two pieces of the policy: the Section 7 rule that charges the fee, and the Section 3 condition that waives it for a wrong item. No configuration ever returned both in its top three.

Retrieval ranks every chunk of the document by relevance and then hands the model only the highest-scoring few (here, three). That cutoff is the top-k.

I classified that failure as a cross-reference limit: the answer needs two distant sections in the same top-k, and no way of cutting the document delivered both. This article is about testing that diagnosis. It did not survive.

The blind spot in the first experiment

The diagnosis was made from a top-3 view, with one retrieval method. That view has a blind spot I did not appreciate at the time: it cannot tell you where the missing chunk actually ranked.

So the follow-up experiment holds chunking fixed, at the section-aware configuration from last time (13 chunks, one per section), and varies only the retrieval method:

• Dense: the same embedding model as before (all-MiniLM-L6-v2), cosine similarity. This arm is a reproduction, and it matched the first experiment's top-three results exactly.
• BM25: keyword matching with two adjustments: rare words count for more, and a word repeated many times counts less each time it repeats. No embeddings involved.
• RRF (reciprocal rank fusion): combines the two rankings by scoring each chunk on one over its position in each list and summing, so a chunk ranked decently by both can outrank one ranked first by only one. No weights or tuning; the constant is 60.

This time I recorded the complete ranking of all 13 chunks for every method and every question, not just the top three. And before running anything, I committed a pre-registration: the parameters, the rules for classifying failures, and a predicted rank for every required chunk under every method. The predictions are in the repository history, in a commit that predates the run. If you have ever caught yourself explaining a result after seeing it, that discipline is the whole reason to bother.

What the full ranking showed

Here is Q3, the question everything failed last time. The rubric (the definition, written before the run, of what counts as a sufficient answer) needs Section 7 plus either Section 3 or the handling table in Section 8. C-3 is the chunk holding Section 3, C-7 Section 7, and so on:

Under the frozen Q3 sufficiency rubric, dense is insufficient at top-3, while BM25 and RRF are sufficient.

But the number that changed my mind about the first experiment is the 5. In the reproduced dense ranking, the waiver section was rank five. Not rank twelve. Not unreachable. It scored 0.6715 against 0.6969 for the chunk at rank three. C-7 held rank 1; ranks 2 through 4 went to Sections 12, 10, and 4. (Ranks three and four were separated by 0.0003. I report that number without leaning on it.)

So the honest correction is this: the first experiment over-attributed Q3, under section-aware chunking, to cross-referencing. With the full ranking visible, the retriever had found real signal for Section 3, and the top-3 cutoff excluded it. What I called a structural limit was, for this retriever on this document, a ranking miss that the diagnostic view was too shallow to see. The cross-reference structure is still real, and it still describes how a human assembles the answer. It just was not the mechanism behind this miss.

That is one diagnosis corrected, not an indictment of the first experiment. The other four questions passed under every method, exactly as they had before.

BM25 found the right chunk, partly for the wrong reasons

The tempting headline is that keyword search understood the question better than embeddings did. The per-term breakdown says otherwise.

BM25 scores are a sum of per-term contributions, so you can decompose exactly why a chunk won. Here is where C-3's winning score actually came from:

Of the four largest contributors shown here, only wrong points directly at the clause that answers the question. The top contributor, because, is a conjunction. I used no stopword list, so nothing told BM25 to ignore it, and in this small corpus it appears in exactly one chunk, which makes it look maximally rare. The question contains "because" as connective tissue ("cancels after shipment because TechNova sent the wrong item"), and the match is pure coincidence of phrasing. sent is worse: it matched a clause about orders sent to the wrong address, a different scenario entirely, that happens to live in the right section.

Nearly half of C-3's score came from because, the, and sent, terms that were either generic or matched the wrong local meaning. BM25 reached the right chunk, and I am glad it did, but if that incidental clause lived one section over, the same mechanics would have sent it there instead. The result is real; the explanation matters more than the scoreboard.

What the fusion result does and does not show

RRF passed Q3 by combining two rankings that disagreed in a useful way: dense put the fee rule first, BM25 put the waiver section first, and the sum carried both into the top three. That is fusion doing its job.

What this run did not demonstrate is the rescue story RRF is often sold on, where fusion promotes a chunk that neither system ranked highly. I checked the recorded rankings for all five questions: every chunk in every RRF top three was already in dense's or BM25's top three. That rescue pattern did not occur here, so this experiment does not provide evidence for it. I also swept RRF's one constant across three orders of magnitude, from k=1 to k=1000, and no verdict changed on any question. On 13 chunks there is simply not much for that constant to do.

How the predictions held up

Ten of twelve predicted ranks were exact, and all fifteen pass/fail predictions were correct. Before that sounds impressive: four of the dense predictions were constrained by the first experiment's published top-3, so they were closer to reading than forecasting. The one genuinely open number was dense's rank for the waiver section, which I predicted at 6. It came in at 5. Wrong, by one, in the direction that made the correction stronger.

What I would change in how I debug retrieval

Three habits, all cheap:

1. Record the full ranking, not just the top-k. The verdict lives in the top-k; the diagnosis lives below it. Rank 5 and rank 13 need different fixes: rank 5 means raising the cutoff or fusing in a second retriever might be enough, while rank 13 means the retriever barely connects the query to the evidence at all, so simply increasing top-k is unlikely to be a useful fix. A top-3 log cannot tell them apart.
2. Ask why a chunk scored before crediting the method. For BM25 the per-term decomposition is a few lines of code, and it is the difference between "keyword search wins on identifiers" and "a conjunction matched by accident." Dense scores do not decompose the same way: cosine similarity is a single number over whole-text embeddings, so there is no equivalent per-term breakdown to inspect.
3. Write your expectations down before you run. My pre-registration is a markdown file and a commit hash. Because it was committed before the run, I cannot quietly rewrite the prediction after seeing the result.

The usual caveat applies, and it is load-bearing: one synthetic policy, 13 chunks, five questions, one embedding model. These results are illustrative, not a benchmark, and the no-stopword tokenizer that handed because its weight was a frozen choice with visible fingerprints on the outcome.

The full rankings, per-term breakdowns, and the pre-registration with its commit history are inspectable in the published experiment and on GitHub.

A miss at top-k tells you the evidence did not arrive. The full ranking tells you whether it was close or nowhere near. Diagnose before you fix.

Evidence & references

• Retrieval diagnosis experiment
• Experiment evidence on GitHub

Previous: Why Fixed-Size Chunking Breaks Retrieval →

This article was originally published on AI in Practice Hub, which hosts the full Retrieval Failure Analysis notes and companion code.