You have a RAG eval suite. Probably faithfulness and answer-relevancy — the two metrics almost every RAG suite starts with. It's green in CI.
Here's a question that suite can't answer for you: if someone weakened your system prompt next week — inverted "do not invent facts," dropped "cite the source," trimmed half the instructions — would those two metrics go red?
I tested it. On the setup below, they didn't.
The setup
A small RAG assistant, graded by a faithfulness judge and a relevancy judge. Standard stuff. I then degraded the prompt one change at a time — the kind of regression a bad edit or prompt drift actually produces — and reran the same suite against each degraded version. This is mutation testing: break the system on purpose, rerun the evals, and count how many breaks the evals catch. The ones they miss are the coverage gaps.
The result — and why it isn't a gotcha
Across 24 injected prompt regressions, the faithfulness + relevancy suite caught zero. Not the inverted "do not invent facts," not the dropped "cite the source," not deleting half the prompt. The faithfulness judge returned a flat top score on every one.
Before you close the tab: this is not "faithfulness is broken," and it's worth being precise about why. Faithfulness measures whether the answer is grounded in the retrieved context — it does not measure whether the answer is correct, and it has no opinion about your prompt. On an answerable case, where the answer sits in the context, a prompt regression that doesn't change the grounded answer is simply invisible to it. The metric is doing its job; its job just isn't "notice prompt regressions." Competent teams already know this and cover it with labeled/correctness evals. The point is that this surfaces the gap on your specific suite, automatically, and names which regressions would walk through.
One caveat that genuinely matters: if your retrieved documents are your source of truth — the docs can't be wrong — then faithful is correct, and this isn't a gap for you. Read every result as per-suite, never universal.
Closing the gap
The fix wasn't a fancier metric. I added one unanswerable case — a question whose answer is not in the context — plus one check: does the assistant abstain ("if it's not in the context, say you don't know") instead of inventing an answer?
That single addition caught exactly the 6 mutants that break the don't-invent guardrail — the inversion, the two operators that delete that line, and both truncations that chop off the prompt tail where it lives. The mutation score moved from 0/24 to 6/24, and each newly-caught mutant was the right one. The remaining survivors are honest, named gaps — citation, tone — each a concrete "write this eval next."
I reran the same experiment against real DeepEval and real Ragas metric objects (not my reimplementation), to check it wasn't an artifact of one rubric set. Same shape: on answerable cases the reference-free metrics sat at the top of their scale on every mutant. It's the metric class, not the library.
What this is, and isn't
- A survivor is a candidate gap you triage — "could this regression happen, and would it be bad?" Some won't survive that question.
- It's a per-suite diagnostic, not a claim about faithfulness in general.
- The mutations are rule-based edits that model real prompt drift; whether they match the regressions your system actually ships is a separate question I'm still working on, and I'll be honest about it in a later part.
If you want to see the loop with no API key, there's an offline demo (mock model) that finds a gap in about a second; the numbers above are a live gpt-4o-mini run, and the configs are in the repo: https://github.com/AshwinUgale/muteval (FINDINGS.md has the reproduction).
The question I'd actually like answered
If you run RAG evals: what's in your suite beyond faithfulness and relevancy? And if you inverted "do not invent facts" in your prompt tonight — would anything in CI go red before a user noticed? I'd genuinely like to hear what people add to cover this.
