I have a confession, and I suspect I'm not alone in it: I've built RAG infrastructure multiple times, and until last week I had never benchmarked any of it.
Unit tests, sure. Integration tests, sure. Everything green, every pipeline connected. But if you'd asked me "is the retrieval actually good?", the honest answer was a shrug with a deployment attached.
For context: I'm building Langhuan, an open-source knowledge base that agents query over MCP. Retrieval is the product. Which made the shrug harder to live with.
The audit that started it
Before writing any eval code, I sat down and listed the retrieval decisions I had shipped. The chunking contract had been through three revisions. Vector and keyword search get fused with RRF. A reranker sits on top. Every one of those decisions had a rationale I could defend in a code review. Not one of them had a number behind it.
I was running on architecture taste. Taste doesn't fail loudly.
What I built
The expensive part of an eval is never the harness, it's the labeled data. I labeled nothing. MIRACL-zh is a Chinese Wikipedia corpus with human-annotated passage relevance, Apache-2.0, so I deterministically sampled 200 real queries from it — same seed, same dataset fingerprint, same exam every time.
Two tracks, because retrieval loses quality at different stages:
- Passage track: ~5,300 single-passage documents. Isolates embedding, FTS, and the fusion itself.
- Long-document track: ~700 full articles through the real pipeline — chunking, parent-child chunks, retrieval.
Each track runs four configurations: vector only, FTS only, hybrid, hybrid + rerank. Metrics are recall@10, MRR@10, nDCG@10 against fixed qrels. No LLM judge anywhere — I wanted the uncertainty of an LLM judge to be the thing this system eliminates.
One rule I'm glad I held onto: determinism. Same fingerprint (dataset, chunker params, models, code version) must produce metrics that match bit for bit across runs, different port, fresh instance and all. If it doesn't, everything you observe is noise and "the change made it better" is folklore.
The smoke test ate first
Step one was a smoke run with a mock embedding — same text always maps to the same vector, zero semantics — to check the harness wasn't biased. Scores hugged the random baseline, so the ruler was straight.
But the smoke run also has to spin up a real standalone instance, create a knowledge base, and write to it. That path returned a 500. Every time. A forward foreign key was missing its deferred check on SQLite — meaning anyone bringing up a fresh instance following my own README would have hit it first thing. The second bug was nastier: vector search had never worked in the production binary at all. The vec extension simply wasn't linked into the build.
Why did my existing tests miss both? Unit tests mock the database. Integration tests run against Postgres. The eval harness was the first thing in the repo to behave like a brand-new user.
The row of zeros
Real model in (bge-m3), full 200 queries, and the results table came back with a row of 0.0000s. The FTS channel. Zero recall. Not "weak on interrogative queries" — zero, across the board.
My first assumption was that the eval was broken. I spent a while proving it wasn't. It was the product.
The kicker sat one column over: hybrid scored 0.9799 — identical to vector-only, digit for digit. My hybrid search had been running as a plain vector search with extra steps, for who knows how long. No errors. No alerts. Users got results, the results looked fine. One of the two channels just... wasn't there.
The bug itself was almost enjoyable
The tokenizer (gse) splits the query 埃及有哪些民族？ — "What ethnic groups does Egypt have?" — into five tokens: 埃及 / 有 / 哪些 / 民族 / ？. And FTS5 matches with AND semantics: a document hits only if it contains every token.
Body text about Egypt will contain 埃及 and 民族. It will never contain 哪些, and it definitely won't contain the question mark. One missing token and the whole query is the empty set. So every query phrased as a question returned nothing, always.
What I like about this bug is that nobody is wrong. The tokenizer segments every word correctly. FTS5 faithfully implements AND. The bug lives in the seam between two correct components — which is exactly the kind of bug component-level tests can never see.
The fix filters the query side: strip punctuation, single-character function words (有, 的, 了), question fillers (哪些, 怎么, 为什么). I kept the stopword list deliberately conservative, because over-filtering silently kills keyword queries — the exact disease being treated. New house rule: touch the list, rerun the eval.
| channel | before | after |
|---|---|---|
| vector only | 0.9799 | 0.9799 |
| FTS only | 0.0000 | 0.1314 |
| hybrid | 0.9799 | 0.9826 |
0.13 looks tiny until you remember what FTS is for: exact strings — file names, model numbers, IDs, proper nouns. Vector search covers "similar meaning"; FTS covers "exactly these characters." Questions get handled by the vector side, keywords are FTS's home turf. Both channels alive is the entire point of hybrid. And 0.9826 > 0.9799 is the first time "hybrid is worth it" was confirmed by my own data instead of by an architecture diagram.
Where the effort actually goes
Two more findings rewired my priorities. Swapping the embedding model (bge-m3 vs Qwen3-Embedding-0.6B) moved recall by 0.4 percentage points — I had spent more agony than that on model choice. Chunk-size experiments? ±0.5pp, not worth changing defaults over. Rerank, meanwhile, actually moved MRR (0.9975 — hits pinned to position one). For the first time the lever hierarchy is measured rather than vibes.
One honesty note: none of these scores are comparable to public leaderboards. My passage track retrieves over a ~5k-passage sample; MIRACL's official numbers are computed over the ~4.9M full pool, and bigger pools mean lower scores. The value is one ruler measuring every change: same fingerprint, diff two metrics.json files, and every gain or loss has a cause.
What I actually got out of it
Not the metrics. For the first time, I feel confident about my own work.
Before this, every statement I made about the retrieval was conditional. "It should handle that." "We use hybrid search" — except functionally, we didn't. Now I know what the system does, where it's strong, where it leaks (about 80% recall on long documents, and I know which failure modes eat the remaining 20%). Every future change — tokenizer tweaks, stopword edits, the traditional-Chinese normalization I have planned — will arrive with a number attached instead of an opinion.
The bugs would still be in there otherwise. That's the part I keep coming back to: two production bugs and an entire missing retrieval mode, and the system looked healthy the entire time. Quietly. If you've shipped RAG more than once and never measured it, you probably have a dead channel you don't know about either.
The harness is a separate binary in the repo (make eval), and the full report with every run and fingerprint is RETRIEVAL_BENCHMARK.md. Steal the approach — it's cheaper than the uncertainty.
