A team builds a retrieval-augmented chatbot over the company's internal policy documents. In the demo, someone asks "how many days of parental leave do we get?" and the bot answers correctly, citing the right PDF. Someone asks about expense limits - correct again. Ten questions, ten good answers, applause, ship it.
Three months later an employee asks whether their contractor status qualifies for the health stipend, gets a confident "yes" assembled from a policy that was superseded last year, and files the claim. Nobody on the team can say when the pipeline started producing answers like that, because nothing was ever measuring whether it did.
That is the real state of most RAG systems in production. The pipeline is not unusually inaccurate - every retrieval system misses sometimes. What's missing is the apparatus that would notice. The demo was treated as the evaluation, and a demo is the one test a RAG system essentially cannot fail: the questions were chosen by the people who built the index, phrased the way the documents phrase things, asked about content everyone knew was in there.
A wrong answer and a right answer look identical
Traditional software fails loudly. A broken API call throws an exception; a bad deploy serves 500 errors; a failing test case turns red. A RAG regression does none of this. Swap the embedding model, change the chunk size, re-index after a document update - the system keeps returning fluent, well-formatted, confidently cited answers. Whether they are grounded answers is invisible in every signal you get: no exception, no latency spike, no schema violation.
This isn't an edge case of RAG engineering; it is a recurring finding in the engineering literature on it. A CAIN 2024 experience report across three RAG case studies (research, education, and biomedical domains) catalogued seven recurring failure points - missing content, missed top-ranked documents, answers retrieved but lost in consolidation, answers present in context but not extracted, wrong format, wrong specificity, incomplete answers - and its two headline takeaways are blunt: "validation of a RAG system is only feasible during operation, and the robustness of a RAG system evolves rather than designed in at the start."
Read that first clause again. You cannot fully validate this class of system before shipping it. Which means the eval harness is not a nice-to-have you add after launch - it is validation-during-operation, systematised: built from the questions real users actually asked, and run continuously as the system and its corpus change. It is the only mechanism by which you ever learn whether the thing works.
"RAG fixes hallucinations" is marketing, and there's a measurement to prove it
The reason teams skip evaluation is an ingrained assumption: retrieval grounds the model, so the hallucination problem is handled. The strongest counterexample comes from a domain with real money behind getting this right. Legal research vendors marketed their RAG products as "eliminating" or "avoid[ing]" hallucinations, even guaranteeing "hallucination-free" citations. When Stanford's RegLab ran the first preregistered empirical evaluation of these tools, the flagship products from LexisNexis and Thomson Reuters "each hallucinate between 17% and 33% of the time."
These are professional-grade systems built by teams with enormous resources over curated, authoritative corpora - the best case for RAG. Retrieval reduced hallucination relative to a bare model, but between one in six and one in three answers still contained fabrication. The gap between "we added retrieval" and "we measured what retrieval actually delivers on our queries" is exactly the gap the vendors' marketing fell into. If they can fall into it, your internal chatbot certainly can.
"Is it accurate?" is two questions wearing one trenchcoat
Suppose you accept the premise and ask: fine, how accurate is my pipeline? The question is underspecified, and that underspecification is why demo-vibes evaluation persists - there was never an agreed definition of correct to check against.
A RAG answer can fail in two independent places, and they need different fixes:
- Retrieval failed. The chunks handed to the model didn't contain the answer - it wasn't indexed, didn't rank into the top-k, or got cut during context assembly. No prompt engineering will fix this; you need to change chunking, embeddings, or ranking.
- Generation failed. The answer was in the retrieved context and the model ignored it, contradicted it, or embellished beyond it. Re-ranking harder won't fix this; you need to change the prompt contract, the model, or add output validation.
An aggregate "accuracy" number collapses these together and leaves you optimising blind. This decomposition is precisely what the RAGAS evaluation framework formalised into separately measurable quantities: context relevance scores the retrieval step (did the right evidence show up, without drowning in irrelevant material?), while faithfulness and answer relevance score the generation step (is every claim in the answer supported by that evidence, and does it address the question?). The Ragas library has since split the retrieval side further into context precision and context recall. The individual metric implementations have known rough edges - but the decomposition is the part that matters, because it turns "the bot was wrong" into a bug report that identifies a component.
What an actual evaluation harness looks like
None of this requires a research team. A minimum viable harness is three artefacts and a habit.
1. A golden set: your domain's definition of correct, written down. Collect 50-100 real questions - from support logs, pilot users, and the subject-matter expert who knows where the knowledge is buried. For each, record the expected answer and which document(s) it must come from:
{
"id": "policy_031",
"question": "Does the health stipend apply to contractors?",
"expected_answer": "No - eligibility requires full-time employment status.",
"must_cite": ["benefits-eligibility-2026.pdf"],
"trap": "superseded 2024 policy still in corpus says yes"
}
That trap field is where the value lives. Easy questions inflate your score; the golden set earns its keep on superseded documents, questions whose answer spans two chunks, questions the corpus cannot answer (the correct behaviour is "I don't know" - the first failure point in the CAIN taxonomy is systems that fabricate rather than decline), and negation cases where the retrieved text says the opposite of what the surface phrasing suggests.
2. Scoring that preserves the retrieval/generation split. Score "did the must-cite document appear in the retrieved set?" separately from "is the answer faithful to what was retrieved?" For the generation side at scale you'll likely use an LLM judge - which is workable but not free of pathology: the MT-Bench study of LLM-as-a-judge found strong judges agree with human raters over 80% of the time, while also documenting systematic verbosity bias (longer answers score better regardless of quality) and signs of self-enhancement bias (judges may favour their own model's outputs). Use a judge, but spot-check it against human labels before trusting it as your regression signal.
3. A regression gate. The harness runs on every change - new embedding model, new chunking strategy, re-index, prompt edit, model version bump - and a score drop blocks the change, the same way a failing test suite blocks a merge. This is the step that converts evaluation from a one-time report into an engineering control. Without the gate, your golden set is a benchmark you ran once in a notebook; with it, "did we just get worse?" has an answer before users provide it.
The demo answered the questions you chose. The golden set answers the ones your users will actually ask - including the ones designed to make your pipeline lie. Build the second thing before you trust the first.
References
- Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., & Abdelrazek, M. (2024). Seven Failure Points When Engineering a Retrieval Augmented Generation System. CAIN '24.
- Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv.
- Magesh, V. et al. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Stanford RegLab.
- Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS.
- Ragas documentation. List of available metrics.
If you want to go deeper than a blog post can, evaluation is where SophiArch's Building AI Applications with LLMs course plants its flag: a full module on golden sets, LLM-as-judge trade-offs, and regression gates, sitting inside a validation architecture that runs from prompt contracts through output checking to production observability.
