Almost every RAG eval writeup I read, including several I have recommended, uses the same mental model. The system is two stages in series. Retrieval either finds the right context or it does not, with probability r. Given the right context, generation either produces the right answer or it does not, with probability g. So end-to-end accuracy is r times g, and if you want to know how good your generator is you divide.
The multiplication is wrong, the division is worse, and the error runs in a consistent direction. It makes your generator look better than it is, and it makes retrieval regressions look like generation regressions.
None of this is deep. It is one term that the product model sets to zero and that is not zero in any system I have measured.
The term everyone drops
The product model says a RAG system answers correctly only when retrieval succeeds. That is not how a language model behaves. It can answer correctly with bad context, because the answer is in its parameters, or because the question is inferable, or because the wrong document happens to contain the right fact. (The parametric-versus-contextual tension has its own literature; Longpre et al., "Entity-Based Knowledge Conflicts in Question Answering", arXiv:2109.05052, is the cleanest construction of it I know.) Call that probability g_bad.
The honest structure is a two-by-two, and end-to-end accuracy is:
e2e = r * g_good + (1 - r) * g_bad
where g_good is P(correct | good context) and g_bad is P(correct | bad context). The product model is the special case g_bad = 0.
If you estimate generation quality by dividing, you get
g_hat = e2e / r = g_good + (1 - r) * g_bad / r
so the bias is exactly (1 - r) * g_bad / r, and it is upward whenever g_bad is above zero. It grows as retrieval gets worse, which is the opposite of reassuring.
Worked, at r = 0.72 and a true g_good of 0.88:
g_bad = 0.00: e2e 0.6336, g_hat 0.8800, bias 0.0000
g_bad = 0.05: e2e 0.6476, g_hat 0.8994, bias +0.0194
g_bad = 0.10: e2e 0.6616, g_hat 0.9189, bias +0.0389
g_bad = 0.20: e2e 0.6896, g_hat 0.9578, bias +0.0778
g_bad = 0.30: e2e 0.7176, g_hat 0.9967, bias +0.1167
g_bad = 0.40: e2e 0.7456, g_hat 1.0356, bias +0.1556
At g_bad = 0.20, which I do not think is high for a general-knowledge domain, you report your generator at 0.958 when it is at 0.880. Just under eight points, in the flattering direction, on a metric people use to decide that generation is fine and retrieval is the problem.
The last row is the useful diagnostic. When g_bad is high enough, g_hat exceeds 1.0. If you have ever divided end-to-end by retrieval recall and got a number above one and quietly clipped it, that clip was the model telling you it was misspecified.
What a single end-to-end number pins down
Bias is the smaller of the two problems. The larger one is that a single end-to-end number does not pin down the system that produced it. You have one equation and, if you count honestly, three unknowns, and note that what follows is the generous version, since I hand you one of them for free.
Fix g_bad at 0.20 and take the end-to-end score from the table above, 0.6896. Here are the (r, g_good) pairs that produce exactly it:
r = 0.55: g_good = 1.0902, impossible
r = 0.65: g_good = 0.9532
r = 0.72: g_good = 0.8800
r = 0.85: g_good = 0.7760
r = 0.95: g_good = 0.7154
Every one of those except the first is a perfectly ordinary system, and they describe completely different engineering situations. At r = 0.65 you have a near-perfect generator starved of context. At r = 0.95 you have excellent retrieval and a generator that fails nearly three times in ten on good context. The end-to-end number is identical.
The impossible row is worth a second look, because it is the one thing a single score does pin down. g_good cannot exceed 1, so r is bounded below by (e2e - g_bad) / (1 - g_bad), which here is 0.612. One number gives you one inequality on one parameter. That is the whole of what it gives you.
So an end-to-end score does not summarise a RAG system's health. It places one constraint on a surface. Watching that constraint move over time tells you the surface shifted, and nothing about which axis it shifted along.
The misattribution, worked
This is the failure I most want people to be able to recognise, because it produces a confident wrong diagnosis rather than an absence of one.
Take a real regression: retrieval degrades from 0.72 to 0.60 because an index rebuild changed chunking. Generation is untouched at g_good = 0.88, g_bad = 0.20.
End-to-end falls from 0.6896 to 0.6080, which is 8.16 points, and you will see that on the dashboard.
Now suppose your retrieval recall figure is stale, as it usually is, because it needs relevance labels and gets recomputed quarterly while end-to-end runs nightly. You divide by 0.72, the number you have.
Your inferred generation quality falls from 0.9578 to 0.8444. That is 11.33 points, and it is entirely an artifact. Generation did not move at all. Worse, the inferred generation drop is larger than the end-to-end drop, so the natural reading is that the generator is the epicentre and end-to-end was partly cushioned.
A team reading that dashboard will spend its next sprint on the generator. I did exactly that in July, in a post about an average that was hiding two failure modes, and I described editing prompts to fix what turned out to be a retrieval gap. This is the mechanism underneath that mistake.
The minimum instrumentation that resolves it
Three quantities, and they have to be measured on the same evaluation set on the same day.
- r, retrieval success. Whatever your definition, hit-rate at k or a graded relevance judgement thresholded, it needs to be recomputed on every run, not quarterly. This is the expensive one because it needs relevance labels, and it is the one whose staleness causes the misattribution above.
- g_good, accuracy restricted to the subset where retrieval succeeded. Conditional, not marginal.
- g_bad, accuracy restricted to the subset where retrieval failed. This is the one nobody computes, it is free once you have the split, and it is the term that makes the arithmetic honest.
It really is a groupby, so here it is rather than described:
import pandas as pd
def decompose(df): # columns: retrieval_ok (bool), correct (bool)
r = df.retrieval_ok.mean()
by = df.groupby("retrieval_ok").correct.agg(["mean", "size"])
return {"r": r,
"g_good": by.loc[True, "mean"], "n_good": int(by.loc[True, "size"]),
"g_bad": by.loc[False, "mean"], "n_bad": int(by.loc[False, "size"]),
"e2e_check": r * by.loc[True, "mean"] + (1 - r) * by.loc[False, "mean"]}
e2e_check is there because it should equal your measured end-to-end score. If it does not, one of the three is being computed on a different slice from the other two, which is a bug I have shipped.
With those three, end-to-end is a derived quantity rather than a measured one, and a movement decomposes into the terms that produced it. One caveat, and it is not small: g_good and g_bad are conditional on a split that retrieval itself defines, so when r moves the two subsets are re-drawn and the conditional numbers move even if the generator is byte-identical. On a two-stratum example I built, and it is the same index rebuild as the section above with the population resolved rather than assumed away: short factual questions that retrieve well and long synthesis questions that retrieve badly. The rebuild hurts the long ones, r drops from 0.720 to 0.600, and measured g_good rises from 0.8894 to 0.9050 with the generator byte-identical. Above I held g_good flat at 0.88 to isolate the division error on its own. In a real rebuild both effects are live at once and they push the inferred number in opposite directions. The effect that motivates this post shows up inside its own fix. So report the three terms with their split sizes beside them, and treat a g_good change that arrives in the same run as an r change as uninterpretable until you re-measure on a fixed question set.
There is one more line worth putting on the same page, because it is the ceiling and it is one multiplication:
r = 0.60, perfect generation on good context, g_bad 0.20: end-to-end cannot exceed 0.6800
r = 0.72: cannot exceed 0.7760
r = 0.85: cannot exceed 0.8800
That is the number to put beside any target somebody proposes, because it is the one the target has to fit under.
FAQ
Is r really binary?
No, and the two-by-two is a simplification I am using because it is the smallest structure that exposes the error. Graded retrieval quality turns the two-by-two into a curve and the bias result above survives, with the sums becoming integrals and the arithmetic becoming less legible. I would rather publish the version people can check by hand.
Does the bias not vanish if retrieval is good?
It shrinks. The bias is (1-r) * g_bad / r, so at r = 0.95 and g_bad = 0.20 it is 0.0105, which is negligible. At r = 0.72 it is 0.0778. The systems that most need a correct diagnosis are exactly the ones where the correction is largest, which is an unhelpful property of the world rather than of the method.
Is g_bad not just measuring that your relevance labels are wrong?
Sometimes, and you should check that first. If g_bad is very high, the likely explanation is that your "retrieval failed" bucket contains documents that were actually sufficient and your labeller disagreed. That is a real and common cause and it is worth ruling out before treating g_bad as parametric knowledge. Either way it is not zero, which is the point.
Why not just use a faithfulness metric on the retrieved context?
Faithfulness measures whether the answer is supported by what was retrieved, which is a different question from whether the answer is correct. A perfectly faithful answer to a wrong document is wrong. Faithfulness is a good instrument for one component of g_good and it says nothing about g_bad, because in the g_bad case the answer is correct while being unsupported by the retrieved context, which faithfulness scores as a failure.
Are r and g_good not correlated? Hard questions fail retrieval and also get answered badly.
They are, and it turns on a definition rather than on an assumption. Write the system as a mixture over question types and the bias is E[(1-r) g_bad] / E[r]. If g_bad is the failure-weighted rate, meaning accuracy on the subset where retrieval actually failed, then E[(1-r) g_bad] = g_bad times E[1-r] by definition, and the whole thing collapses back to (1 - r_bar) g_bad / r_bar. I checked this in exact rational arithmetic over mixtures of two to four strata with deliberately strong dependence between r and both generation rates, and the two sides are the same number rather than two numbers that agree to fifteen places. It is an identity.
The condition is on both conditionals, not just one. g_good has to be the retrieval-success-weighted rate and g_bad the retrieval-failure-weighted rate, which is what steps 2 and 3 above measure. Average the per-type rates any other way and the identity goes: on my four-stratum case, substituting a plain weight-weighted g_bad of 0.187 for the failure-weighted 0.095 turns a true bias of 0.1012 into a computed 0.1986.
Where dependence does bite is if you substitute a closed-book rate for it. At r = 0.72, if the true failure-subset g_bad is 0.20 and you plug in a closed-book number of 0.35, you compute a bias of 0.1361 against a true 0.0778. That is the error, and it is an error of measurement rather than of model.
Open question
The closest thing I have seen published is the no-retrieval ablation most RAG papers report, and that is a different quantity. A closed-book baseline is accuracy with no context. g_bad is accuracy with wrong context, and wrong context is not neutral: Shi and colleagues showed that irrelevant material in the prompt actively degrades accuracy on problems the model otherwise solves (Shi et al., "Large Language Models Can Be Easily Distracted by Irrelevant Context", arXiv:2302.00093, 2023). I would therefore expect g_bad to sit below the closed-book number, though their setting is grade-school arithmetic with an inserted distractor rather than retrieval, and I have not seen no-context and wrong-context measured side by side on the same knowledge task. Nobody reports that gap. Reporting it needs no new labels beyond the ones a retrieval evaluation already requires. It is one groupby.
My suspicion, and it is only that, is that it is much higher than people expect for general-knowledge benchmarks and much lower for genuinely proprietary corpora, and that the gap explains a chunk of why RAG results published on public benchmarks do not reproduce inside companies. On a public benchmark the model may already know the answer, so g_bad is large, the product model's bias is large, and the reported generation quality is inflated. On an internal corpus of last quarter's contracts, g_bad is near zero, the product model is nearly correct, and the same pipeline reports a much worse generator.
That is a testable claim and I do not have the data to test it. If you have both kinds of corpus and the same pipeline over them, the groupby is four lines and I would very much like to see the two numbers side by side.
