TL;DR. Our ticket-routing eval scored 0.94 for five weeks. The number was manufactured. We had built a dynamic few-shot selector that retrieved the eight nearest labeled examples for each input, and we built its index out of the same labeled_tickets.jsonl the eval set was sampled from. So for every eval case, the nearest neighbor in that index was the eval case itself, gold label attached, pasted into the prompt directly above the question we were about to ask. The model was not answering. It was copying. Measured against tickets the index had never seen, real accuracy was about 0.79. The reframing that stuck for me: contamination is not just a training-time problem you inherit from a model vendor. Any pool you draw prompt content from is part of your eval's input. If your eval set and your few-shot pool share a parent file, the leak is yours, in your repo, shipped by someone on your team last Tuesday. Split the pools by content hash at the source, then check the prompt and not only the training data.
The trace that ruined a good number
I was not looking for this. I was chasing p99 latency on the routing endpoint, which had crept past two seconds and was making the queue back up. So I pulled a slow trace and started reading the prompt we actually send, top to bottom, the way you do when you suspect somebody stuffed too much into the context window.
The prompt had a system block, then eight few-shot examples, then the ticket to classify. We rendered them nearest-last, so the closest match sat directly above the question.
Example eight was the ticket to classify. Same text. Same customer. And sitting under it, formatted as the demonstration answer, was the label we were about to grade the model on.
I read it three times. Then I checked four more traces. Same shape every time: the last example before the question was the question.
Our eval had been reporting 0.94 since the selector shipped. Nobody questioned it because 0.94 is a believable number. Five weeks. Nobody asked once. If the suite had printed 1.00, someone would have opened it within the hour, because a perfect score reads as a bug. A 0.94 reads as a good quarter. A total leak still did not mean a perfect score: the other seven demonstrations pulled against the exact match often enough to cost a few points, which is precisely what made 0.94 look earned. It went in a deck. Somebody put it on a slide with an arrow pointing up. The score was high enough to celebrate and low enough to trust, which is the worst place a wrong number can sit.
select_examples()
The mechanism is boring, which is why it survived review.
We started with static few-shot: eight hand-picked examples, hardcoded, same eight for every request. It worked fine and it was obviously fine, because you could read the eight in the diff.
Then someone (me, partly, in a design review I do not get to distance myself from) pointed out that a fixed eight cannot cover billing and abuse reports and integration bugs at once. Retrieve the examples instead. For each incoming ticket, embed it, pull the eight nearest labeled tickets out of the index, put those in the prompt. Better coverage per token. This is standard practice and I still think it is right.
The index got built from labeled_tickets.jsonl, which was where every labeled ticket lived. Roughly 1,900 rows at the time.
The eval set was 600 rows sampled from labeled_tickets.jsonl.
Those two sentences are the whole bug. Both artifacts were correct in isolation. Both were reviewed. Nobody put them side by side, because they lived in different files, owned by different people, merged six weeks apart.
That gap is the part worth generalizing. Neither diff was wrong. A reviewer on the selector PR sees a sensible retrieval change and asks about latency and index freshness. A reviewer on the eval PR sees a defensible sample size and asks whether 600 cases is enough. Both are good questions. Neither reviewer gets asked the only one that mattered, which is whether these two things read the same rows, because that question is not visible in either diff. It lives in the space between them. We had no review process for the space between two files, and I am not sure most teams do.
Here is what the selector did, reduced to the part that matters:
def select_examples(query, index, k=8):
# index was built from every row in labeled_tickets.jsonl
return index.search(embed(query), k=k)
Then the eval harness:
cases = random.sample(load("labeled_tickets.jsonl"), 600)
Read them together and the failure is arithmetic, not machine learning. The eval case is in the index. The eval case's own embedding is its nearest neighbor, at distance zero. Every single one of the 600 eval cases retrieved itself as its own nearest neighbor and carried its gold label into the prompt. Not some of them. All of them. A retriever asked to find the most similar labeled example to a text that is sitting in its own index will return that text, because nothing is more similar to a string than the string.
We also had about 40 exact duplicate tickets in the pool, because customers paste the same complaint twice and support macros generate identical bodies. Those would have leaked across a naive random split even without a retriever. The retriever just made the leak total instead of partial.
Why the retriever hands over the answer
Worth being precise about what broke, because "contamination" gets used loosely enough to stop meaning anything.
The model did not memorize our tickets during pre-training. The vendor did not train on our data. Our fine-tune was clean. Every version of contamination I had read about was about the training corpus, and every one of those was genuinely not our problem.
The leak happened at prompt construction time, in our code, at inference, on every request. The model saw the answer in the line directly above the question, in a block explicitly labeled as examples of correct behavior. It did what any competent few-shot learner does with a demonstration that exactly matches the query. It copied the label.
So the eval was measuring copy fidelity. That is a real capability. It is not the capability we were shipping, and it is not the one the number claimed.
The published work here is almost entirely about training data, and it is worth knowing even though it does not describe this exact bug. Zhou et al., in Don't Make Your LLM an Evaluation Benchmark Cheater, work through how benchmark leakage into training "can dramatically boost the evaluation results," producing an unreliable read on what a model can do. Their setting is pre-training and fine-tuning corpora. Mine was a JSONL file and a vector index.
The mechanism ports anyway. The model cannot tell where in its input a leaked answer came from, and neither can your score. Weights or context window, the arithmetic is the same: if the answer reached the model before the question, the number measures retrieval, not reasoning. The difference is that training contamination is mostly somebody else's to fix, and prompt contamination is entirely yours. I find that clarifying rather than comforting.
contamination_check.py
The check we should have had. Standard library only, so it runs anywhere Python does and there is nothing to install in CI.
Two passes. A normalized hash catches verbatim and cosmetically edited duplicates. An n-gram containment score catches the case where an eval item sits inside a longer example. Containment rather than Jaccard on purpose: a short eval case buried in a long few-shot example still scores 1.0, and that is exactly the leak worth failing on.
"""contamination_check.py: does the few-shot pool already contain the eval answer?"""
import hashlib
import re
import unicodedata
_PUNCT = re.compile(r"[^\w\s]", flags=re.UNICODE)
_WS = re.compile(r"\s+")
def normalize(text):
"""Casefold, strip punctuation, collapse whitespace. Defeats cosmetic edits."""
text = unicodedata.normalize("NFKC", text).casefold()
return _WS.sub(" ", _PUNCT.sub(" ", text)).strip()
def fingerprint(text):
return hashlib.sha256(normalize(text).encode("utf-8")).hexdigest()
def ngrams(text, n):
toks = normalize(text).split()
if len(toks) < n:
return {" ".join(toks)} if toks else set()
return {" ".join(toks[i:i + n]) for i in range(len(toks) - n + 1)}
def containment(case, example, n):
"""Fraction of the eval case's n-grams that also appear in the example."""
gc = ngrams(case, n)
if not gc:
return 0.0
return len(gc & ngrams(example, n)) / len(gc)
def audit(eval_set, fewshot_pool, n=5, threshold=0.6):
"""Return (eval_idx, pool_idx, score, kind) for every eval case the pool leaks."""
by_hash = {}
for i, ex in enumerate(fewshot_pool):
by_hash.setdefault(fingerprint(ex), i)
hits = []
for j, case in enumerate(eval_set):
i = by_hash.get(fingerprint(case))
if i is not None:
hits.append((j, i, 1.0, "exact"))
continue
scored = ((containment(case, ex, n), i) for i, ex in enumerate(fewshot_pool))
score, i = max(scored, default=(0.0, -1))
if score >= threshold:
hits.append((j, i, round(score, 3), f"{n}-gram"))
return hits
if __name__ == "__main__":
fewshot_pool = [
"Escalate to a human when the customer mentions legal action.",
"Refund the order automatically if it shipped more than 30 days ago.",
"When a customer asks to escalate to a human because they mention legal "
"action or a lawsuit, hand the ticket to the on-call agent immediately.",
]
eval_set = [
"Escalate to a human when the customer mentions legal action.", # verbatim
"escalate to a human when the customer mentions LEGAL ACTION!!", # cosmetic edit
"Escalate to a human because they mention legal action or a lawsuit.", # fragment
"Hand off to a person if the buyer threatens to sue.", # paraphrase
"Ask for the order number before checking shipment status.", # clean
]
hits = audit(eval_set, fewshot_pool)
leaked = {j for j, _, _, _ in hits}
for j, i, score, kind in hits:
print(f"LEAK eval[{j}] <- pool[{i}] score={score:<5} via {kind}")
print(f"\n{len(leaked)}/{len(eval_set)} eval cases leaked by the few-shot pool")
for j, case in enumerate(eval_set):
if j not in leaked:
print(f" clean: eval[{j}] {case!r}")
Output:
LEAK eval[0] <- pool[0] score=1.0 via exact
LEAK eval[1] <- pool[0] score=1.0 via exact
LEAK eval[2] <- pool[2] score=1.0 via 5-gram
3/5 eval cases leaked by the few-shot pool
clean: eval[3] 'Hand off to a person if the buyer threatens to sue.'
clean: eval[4] 'Ask for the order number before checking shipment status.'
Three of five caught. The cosmetic edit gets normalized into an exact hit, and the fragment gets caught by containment against the longer pool entry. Both of those would have slipped past a naive set(eval) & set(pool).
Now look at what the script calls clean.
The paraphrase the check calls clean
eval[3] is "Hand off to a person if the buyer threatens to sue." The pool contains "Escalate to a human when the customer mentions legal action." Same rule. Same decision. Zero shared 5-grams, so my check reports it as clean and moves on.
It is not clean. It is the same eval case wearing a different coat, and a model given the pool entry will get eval[3] right for reasons that have nothing to do with understanding your routing policy.
This is a known and documented hole, not something I discovered. Yang, Chiang, Zheng, Gonzalez and Stoica went at it directly in Rethinking Benchmark and Contamination for Language Models with Rephrased Samples. Their finding is that "simple variations of test data (e.g., paraphrasing, translation) can easily bypass these decontamination measures," where the measures in question are exactly the string and n-gram matching my script does. They report that a 13B model can overfit a benchmark and reach performance on par with GPT-4 when rephrased test data is left in, and they found 8% to 18% overlap with HumanEval sitting in pre-training sets like RedPajama-Data-1T and StarCoder-Data. Their decontamination tool is public at lm-sys/llm-decontaminator, Apache-2.0, and it uses an LLM to catch what string matching cannot.
Again their setting is training corpora and mine is a prompt. The blind spot transfers cleanly, because n-grams do not know what a sentence means in either setting.
I have not run their tool against our pool yet, so I am not going to tell you what it would find. What I will say is that the cheap check is worth shipping anyway. It caught three of five in a toy example and it caught our real leak on the first run, because our leak was verbatim. Verbatim is the common case. Paraphrase contamination is real, and my honest position is that I do not currently know how much of it we have, which is a different sentence from "we don't have any."
Where else the same pool leaks
Once I had the shape of it I went looking elsewhere and found three more instances in our own repo within a day. Listing them because the retriever version is the flashiest and probably the least common.
Static few-shot, curated out of the labeled pool. Before the retriever we had eight hardcoded examples, and I had filed them as safe precisely because they were hardcoded. They came from the same file. Two of the eight were in the eval set. A fixed leak is smaller than a total leak. It is still a leak, and it sat in that prompt for months while I told myself static few-shot was the conservative option.
RAG evals where the corpus contains the eval documents. Same arithmetic, different index. If your eval questions were written from documents that live in the retrieval corpus, the retriever will hand the model the exact paragraph each question was written from. This one is arguably fine, because production does the same thing. It stops being fine the second you report the score as evidence about the model rather than about your retriever.
Synthetic eval cases generated from the seed examples. The easiest to walk into and the one I would bet is most widespread right now. You ask a model for 500 eval cases. You seed it with your best labeled examples so the output looks like your domain. Those same examples are in your few-shot block. Your eval set is now a paraphrase of your prompt. No file is shared, no hash collides, and contamination_check.py finds nothing, because there is nothing verbatim left to find. It is the paraphrase case from the Yang et al. paper, arriving through a door we built ourselves and held open.
The common factor is not retrieval. It is one pool of text doing two jobs: teaching the model and grading it. Whenever those roles drink from the same well, the score is compromised, and how badly is not knowable from the score.
Disjoint by construction, not by discipline
Detection is a smoke alarm. It tells you the kitchen is already on fire. The actual fix is to make the overlap impossible to express.
We stopped sampling the eval set and the few-shot index separately from one parent file. Instead, every record gets assigned to exactly one side by hashing its normalized content:
def assign(text, eval_frac=0.35, salt="fewshot-v3"):
h = hashlib.sha256((salt + normalize(text)).encode("utf-8")).hexdigest()
return "eval" if int(h[:8], 16) / 0xFFFFFFFF < eval_frac else "fewshot"
def split_pool(pool, eval_frac=0.35, salt="fewshot-v3"):
buckets = {"eval": [], "fewshot": []}
for rec in pool:
buckets[assign(rec, eval_frac, salt)].append(rec)
return buckets["eval"], buckets["fewshot"]
Three properties earn this over random.sample, and the third is the one that actually mattered to us.
It is deterministic. No seed to forget, no ordering dependency. The same record lands in the same bucket on my laptop, in CI, and in the batch job that rebuilds the index at 4am.
It is stable as the pool grows. Adding 500 new tickets does not reshuffle the existing split, so last month's eval numbers stay comparable to this month's. A reshuffling split quietly moves your baseline and you get to spend a day proving the model did not regress.
It puts duplicates on the same side. This is the part random.sample cannot do. Because the hash runs over the normalized text, our 40-odd exact duplicates and their cosmetic variants all resolve to one bucket. Under random sampling, a duplicated ticket had a real chance of landing one copy in eval and one in few-shot, which is the leak reappearing through the back door after you thought you had closed it.
Run it over 2,000 synthetic records and the realized eval fraction lands within about a point of the 0.35 target, which is the usual hash-bucket wobble and does not matter at this size. The exact counts depend on your strings, so do not pattern-match mine. The part that does not wobble: feed it the escalation rule plus its shouty and double-spaced variants and all three land in the same bucket, every run, on every machine.
The salt is there so you can rotate the split deliberately. Bump fewshot-v3 to fewshot-v4 and you get a fresh partition, on purpose, in a diff, with a name someone has to review.
Then it goes in CI as a gate, not a dashboard. The audit runs on every PR that touches either artifact, and a nonzero hit count fails the build with the offending indices printed. It takes about a second on our pool. I have opinions about slow eval gates, but a hash join over a few thousand strings is not where your merge queue goes to die.
The number after the fix
Rebuilt the index from the few-shot side only. Re-ran the 600-case eval.
0.79.
That is roughly a 15 point drop, and it is the first number that suite ever produced that meant anything. Nobody enjoyed the meeting. The version I would defend now is that we did not lose 15 points, we found out we never had them, and we found out from a trace instead of from a customer.
The follow-on was more interesting than the drop. With a real number, the error slices were legible for the first time. Abuse reports were dragging well below the mean while billing sat above it, and that gap had been invisible at 0.94 because copying the label works equally well for every category. Contamination does not just inflate your score. It flattens it, and a flat score hides the shape of the problem you were trying to see. We spent the next sprint on abuse-report examples and got some of the 15 points back honestly, which took actual work.
What I'd check first
- Print one real prompt from a failing eval run and read it end to end. Not the template. The rendered string, with the retrieved examples in it. If the eval case appears in its own prompt, stop and go fix that before you interpret another score.
- Diff the provenance of the eval set against the few-shot pool. Not the contents, the source. If both trace back to the same file, table, or index, assume overlap until a hash join says otherwise, and treat any shared parent as a leak that has not been found yet.
- Suspect the believable number, not just the perfect one. 1.00 gets investigated. 0.94 gets a slide. Ask what score you would have accepted without checking, and go check that one first.
