I mistook movement for improvement.
For three months, I changed our ranking algorithm every two weeks.
- BM25.
- Hybrid BM25 + vector search.
- A cross-encoder reranker.
- A different embedding model.
- Yet another embedding model.
Every change moved a few documents up or down the list.
None of them made the answers better.
The Same Wrong Answer, Wearing a Different Tie
The user asked:
"Why does the gateway reject service X?"
With BM25, the top result was a glossary page defining gateway. Relevant by keyword. Useless by intent.
With hybrid retrieval, the glossary page dropped to third. A Jira ticket mentioning service X moved to second. It described the symptom, not the cause.
With a cross-encoder reranker, the Jira ticket became the top result. The glossary page fell to fourth. The architecture document that actually explained the exception remained at seventh.
With a different embedding model, that architecture document finally moved to fifth.
I remember thinking we'd finally made progress.
We hadn't.
Every new algorithm felt like progress because something moved.
The problem was that nothing important moved.
The Document That Mattered Never Became a Candidate
The architecture document existed.
I knew it existed.
I had even written part of it.
It never appeared in the top results.
Not because it was ranked poorly.
Because it was never selected.
The ranking algorithm could only sort the candidates it received.
The document that actually explained the answer never entered that candidate set.
No ranking algorithm can promote a document that was excluded before ranking even began.
The Moment It Clicked
I had five search results on my screen.
BM25 returned them in one order.
The cross-encoder returned them in another.
A different embedding model shuffled them again.
They were still the same documents.
Not one of them answered the question.
Only then did I ask myself:
Why am I ranking these documents at all?
The Question That Changed Everything
I stopped asking:
How do I rank these documents better?
I started asking:
Why are these documents candidates in the first place?
That question led me backward.
Not to another ranking algorithm.
But to every previous question I had been trying to answer.
Why didn't a larger context window help?
Why did chunking make retrieval worse?
Why did search keep finding documents but miss the explanation?
I realized those weren't separate problems.
They were different symptoms of the same one.
I had been optimizing the last step of retrieval while the earlier ones were quietly determining the outcome.
The Hierarchy of Retrieval
I now think about retrieval as three different layers.
- Selection — What becomes a candidate at all?
- Assembly — How is information preserved, structured, and connected?
- Ranking — In what order should the candidates appear?
For months, I focused almost entirely on the third layer.
The first layer was already deciding whether the answer had any chance of being found.
Perfect ranking can only reorder what has already been selected.
It cannot retrieve what never became a candidate.
I kept polishing the order of documents that should never have been candidates.
I had mistaken ranking for retrieval.
In the next post, I'll look at a different question: why organizations rarely lose information—they lose the connections that once made that information meaningful.
