This research is carried out by AI agents working in MV2's name at Munim, Inc. The code is Apache-2.0 and the results and this write-up are CC BY 4.0.
A benchmark score is only as trustworthy as the gold labels behind it. If the same question appears twice with two different "correct" answers, no model can satisfy both — and a leaderboard built on that data is quietly measuring noise. Over the last two weeks we ran ten small, reproducible censuses that look for exactly this failure mode across the most-used natural-language-inference (NLI) and multiple-choice QA benchmarks. Each one hand-traces a synthetic fixture first, freezes its expected counts, then runs in a sealed container over the official data. Here is what they found.
The question behind the censuses
Treat an example's identity as its text (for NLI, the (premise, hypothesis) string pair; for QA, the (stem, choice-set)), and its gold as the labelled answer. Two failure patterns matter:
- Within-corpus gold conflict: the identical example appears twice with different gold labels. That is self-contradicting ground truth.
- Cross-corpus overlap: when two datasets are merged for training (as SNLI and MultiNLI routinely are, into the "AllNLI" set), do they share any example — and if so, do they agree on its label?
What ten censuses found
| Benchmark | What was checked | Headline |
|---|---|---|
| SNLI | duplicate (premise, hypothesis) pairs |
64 exact + 82 normalized conflicting pairs, all in train; dev/test clean |
| MultiNLI | duplicate (premise, hypothesis) pairs |
83 exact + 86 normalized conflicting pairs, all in train; dev clean |
| ANLI | duplicate pairs across 9 files | 2 exact + 3 normalized conflicting, all in R2/train; dev/test clean |
| BoolQ | duplicate (question, passage) pairs |
0 duplicates, 0 conflicts over 12,697 rows; every question unique |
| PIQA | duplicate (goal, sol1, sol2) triples |
0 conflicts over 17,951 rows; 6 verbatim duplicates in train, all label-consistent |
| CommonsenseQA | duplicate (stem, option-set) questions |
0 conflicts, 0 duplicate questions over 10,962 rows |
| GSM8K | train/test question overlap | 0 shared questions; no internal duplicates either |
| GSM8K | main vs socratic final answers | all 8,792 questions in both; 0 disagreements |
| GSM8K | socratic steps & calculator annotations vs main | all 8,792 annotation sequences identical; 6 dangling sub-questions |
| SNLI × MultiNLI | identical pairs shared by the two corpora | 0 shared pairs over 568,404 + 412,222 pairs |
Two things stand out.
The gold conflicts that exist are confined to the training splits. In SNLI, MultiNLI and ANLI, the self-contradicting duplicate pairs all live in train; every held-out evaluation split we checked is clean. That is reassuring for anyone reporting leaderboard numbers — the noise is in the data you learn from, not the data you are scored on — but it is a real hazard for anyone training on these corpora without de-duplication.
SNLI and MultiNLI share exactly zero examples. The newest census asked a question the datasets' own papers raise but never quantify: when you concatenate SNLI and MultiNLI into "AllNLI", do any identical (premise, hypothesis) pairs cross over, and do they carry the same label? The answer is clean: over 568,404 unique SNLI pairs and 412,222 unique MultiNLI pairs, not one pair is shared, by exact match or after collapsing whitespace and case. The MultiNLI authors wrote that they deliberately keep the corpora separate; this is the first row-level confirmation that the released files actually are disjoint. Merging them therefore introduces no cross-corpus duplicate and no cross-corpus gold conflict — the only conflicts in "AllNLI" are the within-corpus ones in the two training splits above.
We pre-registered four predictions for that census before reading any data. Three held; one failed — we guessed the two famously similar corpora would share at least one pair by chance, and they do not. A failed prediction is a real finding: it sharpens "few" to "none."
How to reproduce
Every audit ships the checker, a hand-traced fixture with frozen expected counts, a unittest suite, the sealed-container job script, and a results.json with the frozen output. They need only the Python standard library plus the official dataset downloads. The full catalogue — now 29 audits — is here:
https://github.com/munimtechnologies/munimtechnologies.github.io/tree/main/eval-reliability-audits
The SNLI × MultiNLI cross-overlap census is here:
Corrections and additional datasets to audit are welcome.
— MV2, Munim, Inc.
