Building a Production-Grade Hybrid RAG System

I went into this expecting hybrid search to beat plain vector retrieval. That's the conventional wisdom, it's what I'd shipped before, and it's what I planned to write about. On my own labeled eval, plain vector won outright: 100% recall on every retrieval metric, on every chunk size I tested. Hybrid never caught up. It matched vector on page and source recall but consistently trailed on exact-phrase recall.

That's not a story about hybrid being broken. It's a story about what BM25 and vector search each turn out to be good at once you measure them separately instead of assuming an ensemble of the two is free.

Plain vector won outright on every retrieval metric I measured. Hybrid never closed the gap on exact-phrase recall.

That same kind of reversal showed up again later. The pattern I built specifically to reduce hallucination turned out to have the worst reliability of the three I tested, while the simplest one, hypothetical document embeddings, had the best grounding by a wide margin. Neither result is what I expected. Both only surfaced because I built an eval instead of trusting my gut.

This is the same territory as the hybrid retrieval work in an agentic system for a client recently and the Elasticsearch-backed pipeline I built. The difference this time is I measured it properly instead of shipping on instinct, and I’m writing up what that measurement actually found.

TL;DR:

• Plain vector retrieval won outright on a labeled 10-question eval: 100% page, source, and phrase recall. Hybrid matched vector on page and source recall but trailed on exact-phrase recall, a real BM25-vs-vector tradeoff, not a bug.
• Reranking never beat raw vector retrieval, in two separate runs, and cost roughly 3x the latency every time.
• Of three Agentic RAG patterns tested, the one built to reduce hallucination (self-RAG) had a 38% clean run rate. The simplest one (HyDE) had 96% grounding precision. No pattern won on every axis.

Here's what building and actually measuring Agentic RAG taught me.

The setup

The corpus for the chunking and retrieval experiments was architecture documentation and eval-related material. For the agentic-pattern experiments (self-RAG, query decomposition, HyDE) I used a smaller three-document corpus: an internal support-agent guide plus vendor agent-building guides from Anthropic and OpenAI.

Stack: pgvector for dense retrieval on Voyage-3.5 embeddings, Elasticsearch for BM25/sparse retrieval, Cohere for reranking, Tavily as the web-search fallback when self-RAG's local corpus can't answer a question, fixed top_k=5 across the chunking and retriever-strategy experiments.

Chunking: the sweet spot isn't obvious

I tested 256, 512, and 1024 token chunks against the same corpus, fixed k=5.

512 tokens won. It's the best balance of specificity and context for this corpus. 256-token chunks work fine for short, localized definitions, but they can under-context a question and still pull weak, table-of-contents-style chunks that match keywords without answering anything. 1024-token chunks pack more tokens per hit and occasionally surface useful extra context, but they flatten multi-part answers and can drop secondary sources entirely.

The takeaway I keep coming back to: quality is not monotonic in chunk size. With a fixed k=5, bigger chunks just mean a bigger context budget, not a better answer.

One caveat that matters for how much weight to put on this: I didn't have labeled gold answers for this comparison. It's retrieval overlap plus a manual read of answer substance, not formal nDCG or recall against ground truth. A tighter version would score answers against a rubric, or check whether required facts show up in the top-k chunks at all. I did exactly that for the next experiment, which is a big part of why that finding held up.

Hybrid retrieval: the reversal

This is the section where the plan and the data disagreed, and I'm reporting the data.

I built a labeled 10-question retrieval eval, gold pages and gold phrases fixed ahead of time, not judged after the fact, and ran it across three chunk indexes and three retriever strategies (vector, hybrid, ensemble), with reranking on and off.

Plain vector won outright. chunk_512 with vector retrieval, no rerank: 100% source@k, 100% page@k, 100% phrase recall, 1.7 seconds. The best result in the whole eval.

Ensemble added nothing. Concatenating vector and hybrid results matched vector's 100% exactly, because the gold pages were already sitting in the Voyage vector hits. It just added about half a second of latency for zero accuracy gain. A strategy can look more sophisticated on paper and still be dead weight if one of its components isn't contributing anything.

Hybrid held its own on the metrics that matter most, 100% page@k and 100% source@k on every chunk index, tied with vector. Where it consistently gave something up was exact-phrase recall:

Right documents, right pages, every time, for both retrievers. But BM25 and dense retrieval don't rank the same passages inside those documents the same way, and that gap only shows up once you check phrase-level recall instead of stopping at page@k.

One example makes it concrete. On a question about "guardrails," BM25 ranked the OpenAI guide's intro and SDK pages, both of which literally contain the word, while Voyage's vector search pulled the classifier, PII, and moderation definition pages, semantically about guardrails without necessarily using that exact word. That's BM25 doing exactly what BM25 is for, lexical matching, and vector search doing what it's for, semantic matching, diverging on a term that has both a literal sense and a conceptual one. Nothing broken there. Just something to know about a corpus before picking a retriever.

The winner doesn't change. chunk_512 with plain vector, no rerank, is still the best single configuration on this corpus: 100% source, page, and phrase recall, in 1.7 seconds. Hybrid is a close second on two of three metrics and a clear step behind on the third.

Reranking, confirmed with a clean re-run

The first pass at reranking was noisy. Cohere rate limits threw errors on more than half the reranked rows, which meant the "100%" scores on some of those rows were computed only over whatever survived, quietly inflating them. Instead of reporting a number I didn't trust, I found two real bugs in that original run: it had shrunk rerank_top_n to 3 candidates but still scored the result against @5, comparing different result-set sizes, and reranked documents weren't preserving metadata order. I fixed both, held rerank_top_n at 5 to match top_k throughout, and reran paced at 8 seconds between calls, well under the rate limit.

Zero errors this time. Same conclusion, and now I trust it:

Source@k and page@k held at 100% across every configuration regardless of reranking. The one place reranking actually changed the outcome, it made things worse. On chunk_1024, Voyage already had the right page in its top-5, and Cohere's reorder dropped "decentralized" out of the top-5 snippets entirely, turning a correct result into a worse one. Best case in the whole table is a tie. Worst case is measurably worse. Every case costs roughly 3x the retrieval latency.

Reranking a result set that's already correct doesn't help. It's not free, and it won't fix a base retriever that's already broken. Worth checking whether your retrieval actually needs the help before you pay for it.

Self-RAG: the real cost is reliability, not hallucination

All three patterns in this post share the same corrective shell: grade the retrieved documents for relevance, rewrite the query and retry, up to a fixed retry cap, if nothing comes back relevant, generate an answer, then grade the generation itself for support and utility, looping back if that check fails. What differs between self-RAG, query decomposition, and HyDE is only the retrieve step itself, not whether they self-correct.

Self-RAG's retrieve step is the plainest of the three: one retrieval call against the corpus, graded, and retried if needed. I built it as a LangGraph node graph and ran it against 8 threads of real questions.

I expected to be measuring how much this pattern cuts down on hallucination. What I actually measured was reliability, and it wasn't good.

3 of 8 runs completed cleanly (38%). 4 errored outright. 1 was interrupted. Mean utility on the successful runs was 2.67 out of 5. Mean "supported over relevant" ratio, a rough grounding-precision proxy computed from fields the eval was already tracking, came out to 60%. Mean latency on successful runs was 21.0 seconds.

I can't claim this pattern dramatically reduces hallucination from this data. There's no non-self-correcting baseline run on the same questions to compare against, so I don't have the A/B I'd need to back that up. What the data shows clearly is more useful anyway: the pattern built specifically to be more careful was the least dependable one I tested. Half its runs never completed.

The pattern built specifically to be more careful was the least dependable one I tested. Half its runs never completed.

Two examples are worth showing directly instead of summarizing.

The other interesting example is a run that answered "who is the president of the United States" by citing Wikipedia and GovTrack. The mechanism turned out simpler than it looked. Tracing through self_rag.py: retrieve() checks state.iterations. On the first pass it queries Postgres and Voyage against the local agent-guide corpus, which obviously can't answer a current-events question. The relevance grade fails, the graph rewrites the query and increments iterations, and on the second pass retrieve() hard-switches to Tavily web search. It's not a router choosing sources up front, it's the rewrite loop correctly recognizing the local corpus can't help and reaching for the web on retry. Utility still only scored a 1, because even the web-sourced answer wasn't especially well-supported (1 of 2 relevant docs actually backed the claim).

One caveat: "utility" here is a single self-graded score per run, and several questions only ran once or twice. It's a signal, not a benchmark, same standard I'm holding the chunking numbers to above.

Query decomposition: high recall, thinner grounding

Query decomposition swaps in a different retrieve step: it breaks a compound question into sub-queries via the model, then retrieves context for each sub-query in parallel using LangGraph's Send primitive. It runs through the same grade, rewrite, retry shell as self-RAG, retrying by re-decomposing the question if nothing comes back relevant.

5 of 5 runs completed cleanly (100%). Mean utility was 4.20 out of 5, the highest of the three patterns. Mean supported/relevant ratio was 46%. Mean latency was 38.1 seconds, the slowest of the three.

That compound question is the standout example. Decomposition generated four sub-queries and retrieved 40 documents, every single one judged relevant to the original question. It's exactly the kind of question this pattern is built for, and it delivered.

The nuance worth calling out: near-perfect "relevant" judgments across the board, but only 40 to 50% of what's judged relevant actually ends up supporting a claim in the final generated answer. High recall, thinner precision. The pattern is good at finding candidate material and less disciplined about what actually grounds the answer it produces. It also costs the most, pulling 10 to 120 raw documents depending on the question and taking the longest to run. That's a real compute and token cost, not just a quality tradeoff to wave off.

HyDE: fewer documents, better grounding

HyDE (hypothetical document embeddings) generates a hypothetical answer to the question first, embeds that, and uses the resulting embedding to retrieve.

5 of 5 runs completed cleanly (100%). Mean utility was 3.60 out of 5. Mean supported/relevant ratio was 96%. Mean latency was 35.0 seconds.

This is the most interesting single number in the whole dataset. HyDE retrieves far fewer documents than query decomposition (typically 5 to 10, versus 10 to 120), but nearly everything it retrieves as relevant ends up actually supporting the final answer. It's the opposite profile from query decomposition: precision over recall.

One clean example: a question asking the assistant to summarize the OpenAI Agents SDK returned 10 documents, judged 7 relevant, and all 7 actually supported the answer. 100% grounding on that run, off a fraction of query decomposition's retrieval volume.

The actual finding: no pattern wins on every axis

Put the three patterns side by side and the real lesson of this post shows up.

If you only looked at utility, query decomposition wins easily. If you only looked at reliability, self-RAG looks broken. If you cared about grounding precision, which claims are actually backed by retrieved evidence and not just plausible-looking documents, HyDE wins by a wide margin.

None of these are wrong to optimize for. They're different failure modes, and which one you can least afford depends on what you're building. A compliance-sensitive workflow probably cares more about the 96% than the 4.20. A user-facing chat product that needs to always respond probably cares more about the 100% success rate than either. The mistake would be picking a pattern because it has the highest single number on one metric, without knowing what you're trading away to get it.

Two of the shared questions I ran across all three patterns make the tradeoff concrete. On "what are the best frameworks for building AI agents?", self-RAG scored utility 3 with 60% grounding, query decomposition scored 3 with 50% grounding at 55.8 seconds, and HyDE scored the lowest utility (2) while being the most grounded (100%). Utility and grounding are not the same thing, and a single quality score can hide that difference completely.

Takeaways

512-token chunks won on this corpus, and quality wasn't monotonic in chunk size. Bigger isn't automatically better once you're past the point of having enough context.

Plain vector beat both hybrid and ensemble outright on the labeled eval. Hybrid matched vector on page and source recall but consistently trailed on exact-phrase recall, a real, structural tradeoff: BM25 and vector search rank differently on terms that carry both a literal and a conceptual sense, not a flaw in either approach. Conventional wisdom says hybrid should be at least as good as vector alone. Measured properly on this corpus, it wasn't.

Reranking added latency without beating raw vector anywhere in the table, confirmed twice, once noisy and once clean. It's not free, and it doesn't fix a base retriever that's already broken.

No single agentic pattern dominates. Self-RAG's real cost turned out to be reliability, not hallucination. Query decomposition trades grounding precision for recall. HyDE trades raw recall for grounding precision.

The connecting thread across both experiments, and honestly the main thing I'd want a reader to take from this post, is that measuring instead of assuming reversed the expected result twice while building this pipeline. That's a bigger lesson than any individual number here.

Measuring instead of assuming reversed the expected result twice while building this pipeline.

What's still unmeasured, stated plainly rather than buried: I don't have a non-self-correcting baseline to properly test whether self-RAG reduces hallucination, only that it's less reliable. "Utility" is a thin, mostly single-run signal, not a validated metric.

What's next

That last paragraph is basically a preview of the next post. No gold-labeled retrieval eval for the chunking comparison, no proper hallucination A/B for self-RAG. Those are exactly the gaps a real eval harness closes, and that's what I'm building next: DeepEval and RAGAS metrics wired into CI, with threshold gates that fail a build before a quality regression ships. This post found the problems. The next one is about catching them automatically.

The full reusable RAG pipeline, swappable vector/hybrid/ensemble retrieval, the self-RAG/query-decomposition/HyDE graphs, and the eval scripts behind every table above, is on GitHub: ai-engineering-boilerplate.

I build agentic systems that hold up in production, not just in demos. This kind of measurement is part of how I work, not a side project. If you're hiring for Applied AI or FDE roles, my GitHub and background are linked below.

LinkedIn · GitHub · X ·