Your golden dataset is too easy

I spent two posts trying to detect a regression I had planted myself, and failed three separate ways. An LLM judge over a golden dataset: p = 1.000. A judge-free deterministic assertion: p = 1.000. Six label-free signals computed on the same outputs: nothing below p = 0.17.

Three instruments, one answer. At some point the honest move is to stop suspecting the instrument.

The prompt change was always there, and one input tier away it produces an effect the same gate catches easily. On the corpus I had written down it produces none worth measuring. Hold every single thing constant, give the summariser less source to work with, and the gate that read p = 1.000 reads p = 0.0020.

What this is following on from

Part 1 established that the judge in this rig is trustworthy: 16/16 on clean inputs, 5/5 on single-variable corruptions, and a self-consistency check. Part 2 pointed that trusted judge at a regression gate and watched it go blind, then blamed sample size.

The setup has not changed. A Claude Haiku 4.5 summariser writes a two to three sentence summary of each of the 16 published posts on this site, at temperature 0.3, capped at 300 output tokens. A Claude Sonnet 4.5 judge scores each summary against its source for faithfulness at temperature 0, with a strict rubric. The regression is one thing: three guardrail sentences deleted from the summariser's system prompt, exactly what a prompt looks like after somebody tidies it up. Four repeats per arm, so 64 gradings per arm.

Everything below ran against real Bedrock calls in my own account. Rounds 6 and 7 are new here, and all of it is in the repo.

The question production actually asks

A golden dataset gives you the one thing production never does: the right answer, written down in advance. On live traffic you have the input, you have the output, and that is the entire inventory.

So before blaming the dataset I tried the other obvious thing. What can you compute from an (input, output) pair alone, with no labels anywhere?

Round 6 applied all six to the 128 summaries rounds 4 and 5 had already produced. No new Bedrock calls: same outputs, same regression, different question asked of them.

Nothing fired. Every content signal that moved at all pointed the same direction, the regression arm being consistently less grounded than the control, and not one reached significance. The best was trigram grounding at p = 0.168.

The detector that was detecting a markdown heading

Before that null meant anything I had to check the instrument, and it is as well I did.

The first version reported that 96 of 128 summaries contained a fabricated number or entity. Seventy-five percent, against a judge that had just failed exactly one of those same summaries. When your label-free signal and your judge disagree by that margin, the signal is wrong. It was:

• 86 of the hits were the word Summary. The summariser likes to open with a # Summary markdown heading. My entity regex saw a capitalised token absent from the source and called it a fabricated entity. I had built a hallucination detector that was mostly detecting a heading.
• Sentence-initial words. Instead, Rather, Yes, Key.
• Plurals. Macs, LLMs, ACLs flagged against sources that say Mac, LLM, ACL.
• Roundings. One summary said 99%+ cache hit ratios. The source says 99.8% and 99.9%. My matcher saw 99 absent from the source and called it fabricated. It is a true statement, and conservative.

Fixed: strip markdown before extraction, require an entity to actually look like one rather than merely start a sentence, stem plurals, and allow a number to be grounded if the source states something it is a faithful rounding of. The count went from 96 to 11.

The broken version is committed as unlabelled_signals_v1.py so you can run it and get the 96 yourself. It was originally only a claim in a code comment, which is not good enough for a post whose whole argument is that you must read what your signal flagged.

That is the part I would want someone to take from this post even if they skip the rest. A label-free signal is cheap to compute and cheap to get wrong, and there is no judge behind it to catch you. The 96 would have looked like a crisis on a dashboard.

Moving the input instead

Three instruments, three nulls, one dataset.

The 16 posts are dense, tightly-written technical writing that already contains every number a short summary would want. An anti-hallucination guardrail has nothing to suppress on an input that offers no temptation to invent. Delete it and the output barely moves, because the guardrail was not doing any work in the first place.

That is testable. Hold the summariser, both system prompts, the judge, the rubric, the posts and the repeat count fixed, and move exactly one thing: how much of each source the summariser is given.

• 12000 characters. Rounds 4 to 6. Not the whole post: ten of the sixteen are longer than that, up to 29,171 characters, so the baseline tier is already a truncation.
• 2000 characters. Intro plus a section.
• 600 characters. The opening paragraph. The model is asked to summarise a post it has mostly not been shown, and must either hedge or invent.

The judge sees the same truncated source the summariser saw, so nobody is scored for omitting text they were never given.

Only one of those two paths has a model in it. The label-free instrument is a set intersection over a log line.

The result

512 new Bedrock calls, all of them for the two short tiers; the 12000 row is reused from rounds 4 to 6. Both instruments were built before this round and neither is tuned to it.

Two columns for the judge gate because they answer different questions. Fisher exact treats all 64 gradings as independent, which they are not. The clustered column is the exact sign-flip over the 16 posts. Part 2 reported Fisher and only reached for the 16 clusters in its power calculation, which was half the problem. This is the column I am quoting.

The dataset was the problem. The same gate, the same regression, the same judge that read p = 1.000 on full posts reads p = 0.0020 when the source is short. 23 failures out of 64 against a control of 7. It survives Holm correction across the 18 tests in that family at p = 0.033, and it survives dropping the outlier post at p = 0.0039. The prompt change does not manifest on the inputs I had chosen to write down.

That is the finding. It is also the one I was least interested in when I started, which is worth noticing.

The number I wanted to headline, and why I am not

The result I actually wanted was the second column. At 2000 characters the judge over ground truth cannot separate the arms, p = 0.625, and a signal computable on a production log line with no ground truth anywhere separates them at p = 0.020. Same outputs, same n. I had a whole post built around that sentence.

It does not survive its own correction. The family is 18 tests: the five label-free hallucination signals at each of three tiers, plus the three judge gates. Length is in the analysis as a diagnostic, not as a candidate signal, so it sits outside the family. Holm over the 18 leaves the 600-character gate at p = 0.033 and takes trigram grounding at 2000 to p = 0.289. Putting length in as well, at 21 tests, gives 0.037 and 0.328, so nothing here turns on that choice. Bonferroni over just the six signals I searched still gives 0.120. A post that spent Part 2 lecturing about power calculations does not get to correct hard where it kills a result it likes and lightly where it saves one.

Worse, and more instructive: the two signals that do survive Holm are the two I threw away. novel_words at 600 characters comes in at Holm-adjusted p = 0.004, and novel_caps at p = 0.033. Both are confounded, and I will show why below, but the honest summary is that my statistically strongest signals are the ones I have mechanistic reasons to distrust, and my mechanistically cleanest signal does not clear correction.

So the label-free result is a lead worth replicating, not a result. It is directionally right at both short tiers, 14 of 16 posts move the predicted way at 2000 characters, it is not a length artefact, and it is nowhere near significant once you account for how it was found. If you take one number from this post, take the clustered p = 0.0020 from the gate, not p = 0.020 from the signal.

Why trigram grounding, and not the other five

novel_words is confounded by length. It is a raw count, and at 600 characters the unguarded model writes 31% longer summaries than the control, 538 characters against 411, p = 0.0006. More words, more novel words. Normalised by content-word count it becomes grounding, which reads p = 0.060 at 600 and p = 0.115 at 2000. But normalised per 1000 output characters instead, it reads p = 0.010 at 600. Two defensible normalisations, two different answers, which is itself a warning about how much freedom you have when you choose a rate.

novel_caps fires at p = 0.002 and is measuring at least two different things. I read the unguarded arm's full hit list at 600 characters: 40 hits, and every one is an acronym or an inference. AWS twelve times, AI six, LLM/LLMs six, API/APIs six, ARM four. On llm-is-not-a-security-boundary the source says "language model" and the summary says "LLM": abbreviation, not fabrication. On part-4-local-dev-docker the model added "ARM" to a post about the --platform linux/amd64 flag on a Mac, which is correct, useful, and genuinely not in the source. That second kind is interesting rather than wrong, because reaching for outside knowledge is exactly what the deleted guardrail existed to suppress. My favourite is AM, which is the regex catching the clock in "cryptic CloudFormation errors at 2 AM". There is no outright fabrication anywhere in that list, and I am not going to claim one I cannot point at.

novel_numbers never separates the arms. It is exactly zero at 12000 characters. At 2000 it flags four bare digits per arm, p = 1.000, and at 600 it flags eighteen across both arms in the wrong direction, control 11 against regression 7, p = 0.500. The most intuitive hallucination signal, the one everybody reaches for first, has nothing in it at any source length.

trigram_grounding is the cleanest of them. It is the fraction of the summary's own word-trigrams that appear in the source, so it is a rate rather than a count. A rate can still track length, so I checked: at 600 characters the two correlate at r = -0.15 pooled and +0.06 inside the control arm. At 2000 characters it moves from 0.083 to 0.055 and 14 of 16 posts move in the predicted direction, which is p = 0.004 on the sign test alone. Two posts reverse, prompt-caching-bedrock-strands at +0.076 and agentcore-memory-read-after-write at +0.002, and I have no account of the first one.

The likely mechanism: a prompt that forbids adding content also, in practice, pushes the model to reuse the source's wording. The baseline prompt never mentions phrasing, so this is a side effect of the content constraint rather than the thing it asks for. Trigram overlap measures exactly that, and unigram overlap is too coarse to see it.

What I now do

• Treat the golden dataset as a hypothesis about where failure lives, and test it. This is the finding that survived. If your eval corpus is the tidy end of your traffic, a real regression can sit at p = 1.000 in CI and p = 0.0020 one input-difficulty tier away. Truncating your own inputs is a crude but cheap way to find out.
• Validate a label-free signal before trusting it. Print what it flagged and read the list. Mine would have reported a 75% hallucination rate that was mostly a markdown heading.
• Correct for the search. If you try six signals, the one that fires needs the correction, and you have to be willing to publish it after the correction rather than before.
• Prefer rates to counts, and then check the other rate. Every count I tried was confounded by output length. Two reasonable normalisations of the same count disagreed at p = 0.060 and p = 0.010.
• Label-free to compute is not label-free to calibrate. I only know trigram grounding responds to this regression because I ran a controlled comparison against a control arm, which is exactly what you cannot do on live traffic. Deploying it still needs a baseline and a threshold.

The honest caveats

Truncation is a proxy for "traffic you did not anticipate", not the thing itself. It holds domain, topic, style and vocabulary constant and moves only the density of grounding material, which is the cleanest single variable available without leaving the corpus. It does not simulate a novel domain, an adversarial user, or a shifted register.

The 300-token output cap is doing some of the work at 600 characters, and it is all one post. Four unguarded outputs at that tier ran all the way into the cap, up to 1531 characters, two of them cut off mid-sentence, against a control maximum of 505. All four are the same post, year-10-study-system-production-ai-failure-modes, all four of its repeats, and all four were judged FAIL. So part of that headline 23 is one post's instruction-following blowout rather than hallucination. Dropping that post entirely leaves 15 clusters and the gate still reads p = 0.0039, with trigram grounding at p = 0.036.

At 600 characters the control is failing too, 7 times in 64. The guarded system degrades at that tier as well, which is worth knowing before you adopt truncation as a diagnostic: a tier where your control also falls over tells you less about the regression than one where it holds.

"14 of 16 posts" is a 2000-character fact and does not replicate at 600. At 600 it is 9 posts in the predicted direction, 6 against, 1 exactly level, which is chance. The 600 p-value comes from the size of a few large moves, not from agreement across posts.

At the 12000 tier the judge saw more of the post than the summariser did. summarize() sends body[:12000] and judge() sends source[:14000], so on the eight posts longer than 14000 characters the judge held up to 2000 characters the summariser never got. Round 7's 2000 and 600 tiers pass the already-truncated body to both, so they are clean. The bias runs toward the top row's null, not against it, but the three rows are not protocol-identical and I would rather say so.

The cross-tier comparison spans an inference-profile change. Rounds 4 to 6 ran on us. inference profiles and round 7 on global. ones, same model version strings, different routing, because I switched for the cost saving between sittings. Both arms within any one tier always ran on the same profile, so the control-versus-regression comparisons that carry every finding are unaffected. AWS documents routing, monitoring and price differences between profiles and makes no output-equivalence guarantee in either direction, so I would not lean on the trend down the table as if the tiers were perfectly comparable. Worth flagging if you copy the cost saving: global. routes outside the US geography, which matters in a regulated shop.

One corpus, one summariser family, one judge, one rubric, one regression. Sixteen posts written by one person. The mechanism is general enough to be worth checking in your setup. The numbers are mine.

The 64 are 4 repeats over 16 posts, not 64 independent observations. Every label-free p-value above is clustered: an exact sign-flip test enumerating all 2^16 assignments over the posts. The judge gate is reported both ways in the results table, because Fisher exact on the counts is the number Part 2 used and the clustered one is the number I am standing behind. Clustering is not automatically the more conservative choice: across the 15 label-free tests it is stricter in 8, looser in 4 and identical in 3. It is stricter in every test that came near significance, which is the case that matters.

All seven rounds, every script including the broken one, and the raw JSON are in the repo: github.com/rajmurugan01/do-you-trust-it-evals.

Series

This is Part 3 of Evals for Production AI, a series on how you actually know an AI system is good once it is live, not according to the dashboard. Part 1 asked what it takes to trust an LLM judge. Part 2 pointed that judge at a deploy gate and watched it go blind. This one found the blind spot was in the dataset, and killed my preferred explanation on the way.

If you are running an eval corpus you suspect is the tidy end of your traffic, happy to compare notes. Find me on LinkedIn or via rajmurugan.com.

Next in Evals for Production AI: a continuous signal with no labels behind it does not come with a threshold. Where you set the line, what a false page costs, and how to tell a real drop from a Tuesday.