Hierarchical retrieval is supposed to help on long documents. Pick the right chapter, then search inside it. Narrow the haystack, then find the needle.
On my long-book benchmark it came third of five, behind a chapter-summary chain and flat chunk RAG, and only barely ahead of naively grabbing the end of the book.
| method | context precision | context recall |
|---|---|---|
naive_first_context |
0.1475 | 0.1458 |
naive_last_context |
0.4150 | 0.3302 |
flat_chunk_rag |
0.3375 | 0.4302 |
chapter_summary_chain |
0.4000 | 0.4771 |
hierarchical_book_rag |
0.3475 | 0.3365 |
My hypothesis was error compounding: if stage one picks the wrong chapter, every later stage searches the wrong text. That turned out to be true and incomplete — which is worse than being wrong, because it would have sent me to fix the right thing and stop there.
The ablation
Same corpus (240,767 words), same 80 gold questions, five variants, 400 scored rows. Two of the variants are given the correct chapter — diagnostics, not deployable methods — purely to measure how much is lost before versus after chapter selection.
| method | recall | precision | hit@1 | hit@3 | hit@5 |
|---|---|---|---|---|---|
hier_current |
0.3365 | 0.3475 | 0.2000 | 0.3875 | 0.4375 |
hier_no_neighbors |
0.4771 | 0.4000 | 0.2000 | 0.3875 | 0.4375 |
chapter_summary_chain |
0.4771 | 0.4000 | 0.2000 | 0.3875 | 0.4375 |
hier_oracle_chapter |
0.7844 | 0.6796 | 1.0000 | 1.0000 | 1.0000 |
hier_oracle_chapter_neighbors |
0.7688 | 0.6925 | 1.0000 | 1.0000 | 1.0000 |
Deleting a feature closed the entire gap. hier_no_neighbors lands on 0.4771 / 0.4000 — the same numbers as the baseline, to four decimals. The method I thought was structurally worse wasn't worse. It was carrying a stage that was hurting it.
And look at hit@k. Identical across the first three rows. Chapter selection did not change at all between hier_current and hier_no_neighbors — same code. Every point of recall that moved, moved downstream, in the stage that pads results with adjacent chunks.
Counting failures directly
| failure type | count |
|---|---|
neighbor_dilution |
32 |
wrong_chapter |
27 |
right_chapter_wrong_chunk |
7 |
ok |
14 |
My hypothesis accounted for 27 questions. The stage I hadn't suspected accounted for 32. I would have fixed chapter routing, seen a real improvement, and never looked at expansion — because the improvement would have confirmed the theory I walked in with.
Neighbour expansion is not simply bad
With the oracle chapter, adding neighbours moves recall 0.7844 → 0.7688 and precision 0.6796 → 0.6925. Recall down slightly, precision up. When you're already in the right chapter, expansion is roughly a wash and can help precision.
The damage happens when expansion runs on top of an uncertain chapter choice, crowding out the good chunks you did find with text that is only adjacent, not relevant. It is a stage whose value depends on the confidence of the stage above it — exactly the interaction a single end-to-end score cannot show you.
So the finding is not "turn off neighbour expansion." It is treat it as a tunable stage rather than a default, conditioned on router confidence.
The sequence that worked
- Disable each optional stage in turn. Cheap, and it found the larger of my two problems.
- Insert an oracle at each boundary. Not deployable, but it partitions the loss into before/after.
- Only then attribute the failure.
If I had skipped to step three, I'd have published "hierarchical RAG underperforms chapter-summary retrieval on long narrative corpora" — well-supported by my headline numbers, and wrong about the cause.
What this does not show
One private narrative corpus, 80 gold questions written from the corpus rather than by independent annotators. Evidence-term overlap scoring, which is a lightweight audit signal and not full semantic correctness. No confidence intervals — the package didn't compute them and I won't imply precision I didn't measure. The oracle variants read gold labels and are not production-realistic. This is not a universal rule against hierarchical retrieval.
Paper: Diagnosing Hierarchical Retrieval Failure in Long-Document RAG
DOI: 10.5281/zenodo.20692450
