How to Make RAG Show Its Evidence: Claim Verification at 15M-Document Scale

Your retrieval demo is beautiful. You dropped a thousand PDFs into a vector database, wired up an embedding model, and now it answers questions with uncanny confidence. You ship it. Everyone's impressed.

Then the corpus grows. Ten thousand documents. A hundred thousand. A million. Fifteen million.

Nothing crashes. No error in the logs. The latency is fine. The answers still arrive, still confident, still well-written.

They're just wrong more often. And you can't tell which ones.

That's the trap of RAG at scale: it doesn't fail loudly, it fails plausibly. The bigger your corpus, the more ways the system has to retrieve a chunk that looks right and build a fluent, cited-sounding answer on top of it. The failure mode isn't a stack trace. It's a sentence that sounds true.

Here's the part people don't want to hear: this isn't hypothetical, and it isn't solved by a bigger model. In 2025, Stanford's RegLab tested commercial, enterprise-grade legal RAG tools — the paid products built specifically to be trustworthy — and found they still hallucinated in roughly 17–33% of cases (Magesh et al., Journal of Empirical Legal Studies, 2025). One in three to one in six answers, wrong, in a domain where wrong gets people sued. (That number is specific to those legal products and the tested tasks — read it as how hard this is, not a universal failure rate every RAG system will hit.)

RAG reduces hallucination. It does not eliminate it. And at 15M+ documents, the margin gets thinner unless you change the architecture — not the model.

This is the pipeline I'd build for 15 million documents and near-zero confident hallucination. The trick isn't a magic retriever. It's four gates that each catch a different lie, and a system that would genuinely rather abstain than make something up.

To be clear up front, so nobody feels sold: the retrieval and scale pieces are an architecture grounded in published research — not a product I've benchmarked at 15M documents. Every number I quote for those is from a cited paper, flagged as such, and the code is illustrative scaffolding, not a drop-in library. But the one claim I actually care about — that checking support and allowing abstention cuts confident-but-wrong answers — I tested myself on a small labeled set. The measured result, and its limits, is near the end.

The whole pipeline at a glance: corpus prep → index → **Retrieve · Constrain · Verify · Abstain, with the metrics you evaluate it on. The rest of the article unpacks each gate — and the goal is to reduce confident, unsupported claims, not to promise zero errors.

What "near-zero" honestly means. It's a design objective, not a guarantee. Not zero errors — the goal is that zero confident, unsupported claims reach the user, because the system converts what would have been a hallucination into an honest "I couldn't verify this." Some will still slip past an imperfect verifier; the aim is to make that rare, bounded, and observable — not to promise it never happens. That reframe is the whole article.

First, why does RAG hallucinate at all?

If you only remember one thing, make it this: a RAG system hallucinates along three common failure paths — the three this architecture is built to catch — and scaling the corpus makes all three worse. (They aren't the whole taxonomy: a retrieved source can also be outdated, poisoned, mis-parsed, or mis-attributed. Those are real, and they're separate controls — source trust and provenance — which I come back to in "where it breaks.")

The three ways an evidence-grounded answer fails — a similar chunk wins, available evidence goes unused, or missing evidence becomes a guess.

1. It retrieved the wrong evidence.
The answer exists in your corpus, but the retriever pulled a chunk that's similar to the question instead of one that answers it. At 1,000 docs there might be one chunk about "Q2 revenue." At 15 million there are forty thousand chunks that mention revenue, hundreds of near-identical boilerplate paragraphs, and a dozen that look perfect and say the wrong number. More documents means more plausible decoys.

2. It retrieved the right evidence and ignored it.
The correct chunk is sitting in the context window, and the model answers from its own parametric memory anyway — or the chunk got buried in the middle of a long context and the model's attention skated right past it. This is the famous lost in the middle effect (Liu et al., 2023): models reliably use information at the start and end of their context and quietly drop what's in the middle. Stuff 50 chunks in and the answer in position 26 may as well not be there.

3. It retrieved nothing useful and answered anyway.
This is the real hallucination. There is no supporting evidence in the top-k, and instead of saying so, the model does what it was trained to do: produce a fluent, helpful-sounding completion. It fills the gap. In a 15M corpus, plenty of legitimate questions simply have no good answer in the data — and a system with no "I don't know" has no choice but to invent one.

Every technique below exists to kill one of these three. If you can't name which failure a component targets, you're cargo-culting complexity.

The principle: you can't prompt your way out

Before the pipeline, the mindset. "Please only answer using the provided context and don't make things up" is a wish, not a control. It helps a little and fails exactly when you need it most — when the context is wrong, incomplete, or buried.

Near-zero hallucination is a pipeline property, not a prompt. Four gates, in order (the architecture diagram up top is the map):

1. Retrieve — get the right evidence into the context, and be able to prove it's there.
2. Constrain — force the answer to cite that evidence, claim by claim. No citation, no sentence.
3. Verify — independently check that every claim is actually supported by what it cites.
4. Abstain — when support is weak, say so, in calibrated language, instead of guessing.

Retrieve fixes failure #1. Constrain fixes #2. Verify and abstain, together, fix #3 — the dangerous one. Let's build each.

Gate 1 — Retrieve the right evidence (and prove it)

Retrieval is where most of your accuracy is won or lost, and it's the stage that breaks hardest at 15M. Four moves matter.

Deduplicate before you embed — or your index lies to you

At scale, your corpus is full of near-duplicates: the same disclaimer on 200,000 pages, boilerplate repeated across contract versions, articles syndicated to six sites. If you embed all of them, three bad things happen: your index bloats (and so does your bill), retrieval returns ten copies of the same chunk and crowds out diversity, and relevance scores get distorted by frequency.

The standard fix is MinHash + LSH (locality-sensitive hashing): hash each document into a signature, bucket similar signatures, and collapse near-duplicates before they ever reach the embedder. It's cheap, it's old, it's boringly effective, and at 15M docs it's usually the difference between a clean index and a hall of mirrors. Whether you need it, and how aggressive to be, depends on how duplicated your corpus actually is — measure your near-duplicate rate first, and keep version/provenance so dedup doesn't silently merge two documents that genuinely differ.

# Dedup before embedding — MinHash + LSH (datasketch). Streams; never holds 15M in RAM.
from datasketch import MinHash, MinHashLSH

def signature(text, num_perm=128):
    m = MinHash(num_perm=num_perm)
    for token in set(text.lower().split()):      # or k-gram shingles for finer dedup
        m.update(token.encode())
    return m

def dedup(docs, threshold=0.9):
    lsh, kept = MinHashLSH(threshold=threshold, num_perm=128), {}
    for doc_id, text in docs:
        sig = signature(text)
        if lsh.query(sig):                        # a near-duplicate is already kept
            continue
        lsh.insert(doc_id, sig)
        kept[doc_id] = text
    return kept                                   # near-dups collapsed pre-embedding

Chunk with structure, then give every chunk its context back

Naive chunking cuts your documents into 500-token windows and destroys meaning at the seams. A chunk that reads "Revenue grew 3% quarter over quarter" is useless on its own — whose revenue? which quarter? The embedding has no idea, because the sentence was torn out of its document.

The fix that actually moves numbers is contextual retrieval: before embedding each chunk, prepend a short, machine-generated sentence that situates it in its parent document. "This chunk is from ACME Corp's Q2 2026 10-Q, in the section on cloud revenue: Revenue grew 3%..." Now the vector and the keyword index both carry the context.

# Prepend a one-line situating context, and KEEP the source id for later verification
def contextualize(chunk, doc_meta, llm):
    prefix = llm(
        f"In one sentence, situate this passage in its document "
        f"(title: {doc_meta['title']}, section: {doc_meta['section']}).\n\n{chunk.text}"
    )
    return {
        "id": chunk.id,                               # provenance → rides into the citation
        "doc_id": doc_meta["doc_id"],
        "embed_text": f"{prefix}\n\n{chunk.text}",    # what you embed AND BM25-index
        "source_text": chunk.text,                    # what Gate 3 later checks the claim against
    }
# tip: prompt-cache the full document so the per-chunk context call stays cheap.

Context makes a chunk retrievable; the preserved source ID and document provenance make it verifiable later.

This is the one architecture choice with clean, published proof. Anthropic measured the share of queries where the right chunk wasn't in the top 20 (Contextual Retrieval, Sept 2024):

A preprocessing step that turns a 5.7% miss rate into 1.9% is one of the best returns in the whole stack. Honest caveat: those are the best-performing embedding configuration in Anthropic's test, not an average across every model — treat it as "defaults are leaving a lot on the table," not "1.9% guaranteed on your data."

Hybrid search: meaning and the literal token

Dense vector search is brilliant at meaning — it knows "cancel my plan" ≈ "terminate subscription." It's terrible at exact strings: error code E-4021, a rare surname, a part number, a statute citation. Those are the queries where a semantic embedding smooths away the one token that matters.

So run both. Dense (vector) search for meaning, sparse (BM25) search for the literal token, and fuse the two rankings with Reciprocal Rank Fusion (RRF) — a dead-simple formula that scores each document by summing 1 / (k + rank) across the lists it appears in:

# RRF: robust rank fusion, no tuning or training. k=60 is a common default — tune it.
def rrf(dense_ranking, sparse_ranking, k=60):
    scores = {}
    for ranking in (dense_ranking, sparse_ranking):
        for rank, doc_id in enumerate(ranking, start=1):   # 1-based ranks (standard RRF)
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

Dense catches the paraphrase. BM25 catches the exact code. RRF lets a document that both rank reasonably well float to the top. This single change kills a huge share of "retrieved the wrong evidence" failures at scale.

def hybrid_retrieve(query, dense_index, bm25_index, n=150):
    dense  = dense_index.search(embed(query), k=n)    # semantic similarity
    sparse = bm25_index.search(query, k=n)            # exact tokens / codes / names
    fused  = rrf([d.id for d in dense], [s.id for s in sparse])
    return [load_chunk(i) for i in fused[:n]]         # ~150 candidates, over-fetched

Over-fetch, then rerank down

Your first-pass retrieval is tuned for speed over a 15M index, so it's approximate — it casts a wide, slightly sloppy net. So cast it wide on purpose: pull ~150 candidates, then hand them to a cross-encoder reranker that reads the query and each chunk together and re-scores them, keeping the best ~20.

The reranker is slow and precise where retrieval is fast and fuzzy. Running it on 150 candidates (not 15 million) is affordable, and it's the step that earns the last chunk of that 5.7%→1.9% drop in the table above.

from sentence_transformers import CrossEncoder
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")    # reads (query, chunk) together

def rerank(query, candidates, keep=20):
    scores = reranker.predict([(query, c["source_text"]) for c in candidates])
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [c for c, _ in ranked[:keep]]              # 150 -> 20 (tune on your eval set)

Retrieve broadly, rerank narrowly. The candidate counts are illustrative — tune them on your own labeled set. Reranking surfaces promising evidence; it does not prove a claim is supported.

The 15M scale note nobody mentions. Fifteen million vectors at 1024 float32 dimensions is ~61 GB of raw vectors — before the index overhead. You will not hold that in RAM naively. Two levers: an on-disk ANN index (HNSW or IVF-PQ) so the index lives on SSD, and embedding quantization (float32 → int8, or even binary) which cuts memory 4–32× for a recall hit you must measure on your own corpus and embedding model — it's often small for int8, but "small" is not a given, and binary quantization in particular can cost real recall. Scale is a memory-layout problem as much as a modeling one.

And keep three kinds of evidence separate, because they get conflated constantly: an index benchmark (ANN latency/recall on 15M synthetic vectors) is not the same as end-to-end retrieval recall on your real, messy documents, which is not the same as extrapolated capacity. A synthetic index proves the search scales; it says nothing about whether the right chunk actually comes back on your corpus. And mind the units: 15 million documents is not 15 million *vectors* — one document becomes dozens of chunks, so a 15M-document corpus is hundreds of millions of vectors. The 61 GB figure above is for 15M vectors; state documents, chunks, and vectors separately, and treat any specific index size as a target configuration unless you've actually built and measured it.

Which vector store survives 15 million vectors?

"Use a vector database" is where most tutorials wave their hands and name whatever they happened to use. At 15M the choice is a real engineering decision — on-disk vs in-memory, quantization support, and whether sparse (BM25) lives in the same system or a second one you have to keep in sync. My honest read of the landscape:

There's no single right answer — the point is that at 15M, "dense + sparse in one system, on disk, with quantization" is the requirement, and you pick the store that gives you those three without a second sync pipeline. I reach for Qdrant self-hosted; a Postgres shop should try pgvector before adding infrastructure.

Gate 2 — Constrain generation: cite, or don't speak

You've got 20 excellent chunks in context. Failure #2 says the model might still ignore them. Gate 2 makes ignoring them structurally hard.

Make citations mandatory, per claim

Don't ask the model to "use the context." Require it to attach a citation — a chunk ID — to every factual sentence it writes, and instruct it that any claim it cannot cite must be dropped or flagged as uncertain. Force the output into a shape like:

{
  "answer": [
    {"claim": "ACME's Q2 2026 cloud revenue grew 3% QoQ.", "cites": ["chunk_8842"]},
    {"claim": "Growth was driven by enterprise seat expansion.", "cites": ["chunk_8847","chunk_8851"]}
  ],
  "unsupported": []
}

This does two things. It drags the model's attention onto the evidence (it has to point at a chunk for each sentence), and it gives Gate 3 something concrete to check. A claim with no citation is a claim the model made up — and now you can see it, instead of it hiding inside a fluent paragraph.

One guard you must not skip: the model can emit perfectly-shaped JSON citing a chunk id it never actually retrieved. Right shape ≠ real citation. Validate every cited id against the retrieved set server-side, before the claim is trusted — a citation is only real if it points at a chunk you actually have in hand.

# A citation is real only if it points at a retrieved chunk. Never trust the JSON shape.
def validate_citations(answer, retrieved_ids):
    valid = set(retrieved_ids)
    for item in answer["answer"]:
        item["cites"] = [cid for cid in item["cites"] if cid in valid]
    # a claim whose citations were all fabricated has nothing left — drop it
    answer["answer"] = [it for it in answer["answer"] if it["cites"]]
    return answer

CITED_SYSTEM = """Answer ONLY from the numbered context below.
Return JSON: {"answer": [{"claim": "...", "cites": ["<chunk_id>"]}], "unsupported": []}
Every claim must cite a chunk id that FULLY states it. If you cannot support a
sentence from the context, do not write it — name the topic in "unsupported" instead.
Never cite a chunk that does not state the claim."""

def generate_cited(question, chunks, llm):
    context = "\n\n".join(f"[{c['id']}] {c['source_text']}" for c in chunks)
    raw = llm(system=CITED_SYSTEM, user=f"{context}\n\nQuestion: {question}")
    return json.loads(raw)        # {"answer": [{claim, cites}], "unsupported": [...]}

Beat "lost in the middle" with order and brevity

Remember failure #2: the right chunk gets buried mid-context. Two defenses:

• Keep k small. This is why you reranked 150 down to 20 (and often down to 5–8 for the final prompt). Fewer, better chunks means less middle to get lost in.
• Order for attention. Place the highest-scoring chunks at the start and end of the context, not the middle, because that's where models actually read (Liu et al., 2023).

A qualitative pattern, not a law — the effect varies by model, task, and context length. Treat evidence ordering as something to test on your target model, not a fixed rule.

Honest nuance: a 2025 replication found the lost-in-the-middle effect isn't universal — it weakens as inputs approach the model's full context window, and long-context studies don't always reproduce it (arXiv:2508.07479). So treat ordering as cheap insurance, not a law. The free win is still "fewer, better chunks."

Route and decompose before you retrieve

One retrieval pass can't serve every question. "What's ACME's refund policy?" is a lookup. "How did ACME's margins change across the last three quarters and why?" is a multi-hop question that no single chunk answers.

So classify the question first, then:

• Simple lookup → one hybrid retrieval, answer.
• Multi-hop / comparative → decompose into sub-questions, retrieve evidence for each separately, then synthesize. Each sub-answer carries its own citations.

Decomposition directly prevents a nasty failure: answering a three-part question from one chunk that happened to rank first. Give each part its own evidence or don't claim it.

Gate 3 — Verify: grade the claims, not the vibes

Here's the uncomfortable truth about Gate 2: a model told to cite will sometimes cite a chunk that doesn't actually support the claim. It'll attach chunk_8842 to a sentence that chunk never said. Citations are necessary, not sufficient. You have to check them.

Citations are necessary, not sufficient — a claim can cite a real passage that doesn't actually entail it. Check each claim against what it points to; keep the supported ones, drop or abstain on the rest.

The move is claim-level faithfulness verification:

1. Decompose the generated answer into atomic claims (one assertion each).
2. For each claim, pull the chunks it cited.
3. Ask a verifier: is this claim entailed by these chunks? (an NLI model, or an LLM-as-judge prompted narrowly for entailment, not helpfulness).
4. A claim that fails entailment is unsupported — strip it, or send the whole answer to the abstain gate.

# Check each claim against ONLY the chunks it cited — entailment, not helpfulness
NLI = ("Does the EVIDENCE fully entail the CLAIM? "
       "Reply SUPPORTED, NOT_ENTAILED, or CONTRADICTED.\nCLAIM: {c}\nEVIDENCE: {e}")

def verify(answer, chunks_by_id, judge):
    kept, rejected = [], []
    for item in answer["answer"]:
        evidence = "\n".join(chunks_by_id[cid]["source_text"] for cid in item["cites"])
        verdict = judge(NLI.format(c=item["claim"], e=evidence)).strip()
        (kept if verdict == "SUPPORTED" else rejected).append(item)
    return kept, rejected          # rejected → stripped, or trigger abstention (Gate 4)

Why atomic and claim-level? Because answer-level scores hide per-claim lies. An answer with nine supported claims and one fabricated one scores 0.9 on a naive groundedness metric and ships the fabrication. Grade each claim and the fabrication has nowhere to hide. This is exactly the direction the measurement research has moved — the RAGAS framework formalized faithfulness as a first-class metric (Es et al., 2023), and 2025 work like Vectara's FaithJudge builds leaderboards specifically for detecting unsupported content (arXiv:2505.04847).

Keep yourself honest: the verifier is itself a model, and FaithJudge's own finding is that detecting unsupported content is still hard — judges miss things. Verification reduces hallucination that reaches the user; it doesn't zero it. Anyone selling you "100% faithful" is selling you the same overconfidence you're trying to remove.

This is also why the Self-RAG and Corrective-RAG lines of work exist — they bake the check into the loop. Self-RAG (Asai et al., 2023) trains the model to emit reflection tokens that judge relevance and support as it generates. Corrective RAG / CRAG (Yan et al., 2024) puts a lightweight evaluator outside the model that scores retrieved docs and, on a bad score, triggers a fallback (like a web search) instead of plowing ahead. Both are the same instinct as Gate 3: don't trust the first draft of the evidence.

Gate 4 — Abstain: the feature everyone treats as a bug

This is the gate that actually gets you to near-zero, and it's the one most teams refuse to build because it feels like admitting defeat.

When retrieval confidence is low, or Gate 3 strips too many claims, or the remaining support is thin — the system says so. "I don't have enough reliable information to answer that." Not a guess dressed as an answer. An honest, calibrated abstention.

The right way to think about this is a 2×2 that keeps two different questions apart: is there sufficient evidence? (answerable vs unanswerable) and did the system answer? (answer vs abstain). Those are not the same axis, and conflating them is how people fool themselves.

Separate "is there evidence?" from "did it answer?" The dangerous cell is unanswerable × answered — a confident unsupported answer. Abstaining when evidence exists is only a missed opportunity; abstaining when it doesn't is a safe refusal. Measure coverage and risk together.

The naive version is a similarity threshold ("if top chunk score < 0.7, abstain"). It works a little and is impossible to tune across question types. The rigorous version is calibrated, risk-controlled abstention:

• Conformal abstention (Abbasi-Yadkori et al., NeurIPS 2024 workshop) samples several answers, measures how much the model agrees with itself, and uses conformal prediction to control the error rate. But read the guarantee precisely: it holds under its sampling assumptions and a defined calibration set, and it bounds the rate of the specific quantity you calibrated on — it is not a switch that magically pins "any real-world hallucination rate" to a number you type in. It's the rigorous target to aim at, with conditions, not a free guarantee.
• The generalization everyone is reaching for is to fuse several uncertainty signals — sequence likelihood, self-consistency across samples, retrieval score, and verifier feedback — into one calibrated probability of correctness, and refuse when it can't clear the budget. This is an active, fast-moving research area in 2025–26, so treat specific framework names as preprints-in-flux and lean on the conformal result above as the load-bearing proof.

The engineering version you can ship today: combine (a) retrieval score, (b) fraction of claims that survived Gate 3, and (c) answer self-consistency across 2–3 samples, into a single confidence, and abstain below a threshold you calibrate on a labeled held-out set so the threshold means something. Be honest about what this is: calibrated thresholding is what you actually ship; the conformal machinery above is the rigorous upgrade once you have the calibration data and are willing to defend its assumptions. Don't call a tuned threshold a "guarantee."

def decide(kept, rejected, retrieval_score, samples, tau):
    # tau is CALIBRATED on a labeled held-out set — not guessed
    support_ratio = len(kept) / max(1, len(kept) + len(rejected))
    consistency   = agreement(samples)            # do 2-3 resampled answers agree?
    confidence = 0.4*retrieval_score + 0.4*support_ratio + 0.2*consistency
    if not kept or confidence < tau:
        return {"action": "abstain",
                "message": "I couldn't verify this from the available sources."}
    return {"action": "answer", "claims": kept}

Here's the reframe that makes abstention worth it:

In a consumer search toy, a wrong answer is annoying. In law, medicine, or finance, a confident wrong answer is the catastrophe — it's the 17–33% that got Stanford's attention. An abstention, by contrast, is just a routing decision: escalate to a human, trigger a deeper search, or tell the user honestly. An abstention is a lead. A confident lie is a liability.

A RAG system that can't say "I don't know" isn't trustworthy at any scale. It's just lucky until it isn't.

The whole pipeline, and what each gate catches

Run a question through all four and what should reach the user are claims that were retrieved, cited, and checked — or an honest abstention. The would-be hallucinations are meant to be caught and converted into "I couldn't verify this." The honest asterisk: the verifier is a model, so some unsupported claims still slip through and some supported ones get wrongly rejected. That residual is exactly the thing you measure (next section) — not a number you get to assume is zero.

That's the aim: not zero mistakes, but driving confident, unsupported claims toward zero, with the leftover rate measured and reported rather than promised.

Prove it — the evaluation you actually run

A pipeline that claims "near-zero hallucination" with no measurement is just a nicer-sounding hallucination. Measure four things, on a labeled eval set built from your corpus:

1. Retrieval failure rate @k — share of questions where the gold chunk isn't in the top-k. This is the number contextual retrieval moves. Track it before and after every retrieval change.
2. Claim support — and measure it two different ways, because they're not the same thing. Context faithfulness (e.g. RAGAS faithfulness): is each claim supported by the retrieved context as a whole? Citation entailment (what Gate 3 actually does): does the specific passage the claim cites support it? A claim can be context-faithful — true given everything retrieved — yet cite the wrong chunk, which is a real defect a context-only score misses. Measure both, and grade claims, never whole answers.
3. Context precision / recall — did the right chunks get retrieved (recall) and were they near the top (precision)?
4. Abstention calibration — the one everyone skips. When the system answers, how often is it right? When it abstains, how often would it have been wrong? A good system abstains almost exactly on the questions it would have botched. If it abstains on easy questions or answers the hard ones confidently, your threshold is miscalibrated.

And two disciplines that separate a credible writeup from a nice-sounding one:

Run it as an ablation, not a single score. One number ("0.9 faithfulness") tells you nothing about which gate earned it. Measure the same eval set three ways — (1) baseline RAG, (2) + claim verification, (3) + abstention — holding the corpus, questions, generator, and criteria constant. For each, report answer coverage (what fraction it chose to answer) alongside answer risk (error rate among the answers it gave). This matters because low risk achieved by refusing almost everything is worthless: coverage and risk are only meaningful together. That ablation, not a diagram, is what proves a control actually did something.

def coverage_and_risk(results):
    # results: [{"answered": bool, "correct": bool | None}]  (correct=None when abstained)
    answered = [r for r in results if r["answered"]]
    coverage = len(answered) / len(results)                        # how much it answered
    risk = sum(not r["correct"] for r in answered) / max(1, len(answered))  # error rate among answers
    return coverage, risk          # report BOTH — low risk at 3% coverage is worthless

Measure the verifier itself, not just the system. Your Gate-3 checker is a model with its own error rate, and a single AUROC doesn't certify anything. Report its precision/recall at the threshold you actually deploy, its false-accept rate (unsupported claims it waved through) and false-reject rate (supported claims it killed), and hand-review a sample of its calls. Crucially, test it on realistic failures — subtly-wrong, plausibly-cited claims — not just cartoon false premises like "which language did Newton invent." The easy cases flatter the verifier; the subtle ones are the ones that ship.

The sample-size trap. "Zero hallucinations in my 20 test questions" is not evidence of near-zero. To put a 95% one-sided upper bound below ~2% you need on the order of 150 clean cases with zero failures — and more the moment one failure appears. Report the interval, not the point estimate.

I ran the one claim that matters — verify + abstain, measured

The full architecture is a proposal. But its core bet — that checking support and letting the system abstain actually cuts confident-but-wrong answers — is testable without a 15M-document cluster. So I tested it.

Setup (deliberately small and scoped). SQuAD v2 is the right dataset because it uniquely ships both answerable questions (the answer is in the passage) and unanswerable ones (it isn't) — exactly the labels you need to measure abstention. I used each question's gold passage as the "retrieved evidence" (this isolates the verify/abstain layer from retrieval, on purpose), an 81M-param extractive QA model (tinyroberta-squad2) as both generator and verifier via its no-answer score, a calibration split to freeze the abstention threshold, and a held-out test set of 1,200 questions (585 unanswerable / 615 answerable, seed 42).

The trade, in numbers: turning "always answer" into "verify, then abstain when unsupported" drops the unsupported-answer rate on unanswerable questions from 100% → 11.6% — an ~88-point fall with non-overlapping 95% CIs — while sacrificing ~20 points of coverage on genuinely answerable questions (100% → 80.3%). That is the whole thesis, measured: you buy a large drop in confident fabrication with a real, bounded loss of coverage.

Two honest findings, not one. Most of the win comes from having any verify/abstain layer at all (unsupported 100% → 18.3%). Calibration on top is a modest incremental gain (18.3% → 11.6%; overall risk 20.6% → 15.3%), bought with coverage on answerable questions (88.5% → 80.3%). I report it as incremental, because it is.

What this does and doesn't show. It demonstrates the mechanism — verification + abstention reduces unsupported answers at a measurable coverage cost — on a small, single-model, single-dataset, gold-context setup. It does not show end-to-end quality at 15M documents; it uses an extractive QA model as a stand-in for a generative one; the "verifier" is the model's own no-answer head, not an independent checker; and SQuAD v2's unanswerables aren't every kind of real-world unsupported answer. The confidence intervals cover sampling noise only, not model or dataset variation. Read it as evidence the control works — not a promise your corpus gets these exact numbers, which is precisely why the evaluation section above matters.

95% Wilson CIs — unsupported rate: baseline 100% [99.3, 100]; +verify 18.3% [15.4, 21.6]; +calibrated 11.6% [9.3, 14.5].

Where it still breaks (because it does)

I'd be lying — in an article about not lying — if I pretended this is bulletproof. The honest edges:

• The verifier is a model. It misses unsupported claims; FaithJudge says so plainly. Gate 3 reduces, never eliminates.
• Evidence can be obfuscated. A claim can be technically entailed by a chunk that is itself wrong, outdated, or adversarially planted. Faithfulness to a bad source is still a wrong answer. Retrieval quality and source trust are upstream problems this pipeline assumes you've handled.
• Multi-hop beyond your decomposition still fails. If a question needs a reasoning chain your router didn't anticipate, no single retrieval serves it.
• Cost and latency are real. Reranking 150 candidates, decomposing questions, sampling for self-consistency, and running a verifier is several model calls per query. Near-zero hallucination is not free; it's a budget you spend deliberately on the questions that warrant it (route the cheap ones through a lighter path).
• Lost-in-the-middle is conditional. The 2025 replication means "order your chunks" is insurance, not a guarantee — don't over-engineer around an effect that may not fire on your model and context length.

None of these are reasons not to build the four gates. They're reasons to keep the abstain gate honest and the eval set growing.

The one idea to take with you

Most RAG advice is about making the retriever smarter. That helps, and Gate 1 matters. But the reason RAG lies at scale isn't that the retriever is dumb — it's that the generator will always, cheerfully, fill a gap with something plausible unless you build a system that catches it and a system that's allowed to say no.

Retrieve the right evidence. Constrain the answer to cite it. Verify every claim against what it cites. And when the evidence isn't there, abstain — out loud, in calibrated language, as a first-class output.

A RAG system you can trust at 15 million documents isn't the one with the best answers. It's the one that knows the difference between an answer and a guess — and refuses to hand you the guess with a straight face.

If this was useful, follow me — I write about building AI systems that are honest about what they don't know. Next up: measuring abstention calibration without a labeled set.

Your turn: what's your current move when retrieval comes back empty — abstain, fall back to the base model, or trigger a web search? And has anyone actually shipped a RAG system that says "I don't know" in production without users hating it? I genuinely want to know. 👇

Sources (each with its own caveats noted inline): Anthropic, "Introducing Contextual Retrieval" (Sept 2024); Liu et al., "Lost in the Middle" (arXiv:2307.03172) and the 2025 replication (arXiv:2508.07479); Magesh et al., legal-RAG hallucination study (Stanford RegLab, J. Empirical Legal Studies 2025); Es et al., RAGAS (arXiv:2309.15217); Vectara FaithJudge — Tamber et al., EMNLP 2025 Industry (arXiv:2505.04847); Asai et al., Self-RAG (arXiv:2310.11511); Yan et al., Corrective RAG (arXiv:2401.15884); Abbasi-Yadkori et al., Conformal Abstention (NeurIPS 2024). The legal-RAG figures were tested on May 2024 versions of the products — directionally still valid, but newer releases may differ. Secondary figures (the 17–33% legal number, the contextual-retrieval table) are reported from the sources above, not independently reproduced here.