Your LLM Judge Is Lying to You, a Little

Judge-only intervals miss the full-human value 18 of 21 times. PPI++ saves ~9% of human labels; disagreement routing costs 2.6x — pre-registered and coverage-gated.

Scope note, first paragraph as promised: 7 English-heavy annotation tasks, 3 mid-tier judges (2 local, 1 API), one run each, $0.29 API spend. Numbers below are scoped to the pre-registered study (846 configs × 2,000 replicates).

Your LLM judge says the system scores 72%. The true number — what full human
labeling would conclude — is 65%. And the ±1% interval you computed describes
the judge's opinions, not the truth. I measured exactly this: judge-only
intervals missed the full-human value in 18 of 21 task-and-panel
combinations across 7 real annotation tasks.

The usual fix — "label a random 100 by hand to check" — is ad hoc. Nobody
knows whether 100 is enough, which 100 to pick, or how to combine them with
the 5,000 judge labels. FewHumans is a library that answers all three
with valid statistics: it routes which items humans label, combines the two
sources with prediction-powered inference, and tells you when to stop looking
at anytime-valid intervals.

Setup

Three judges (two local Qwen 9B/4B through Ollama, one gpt-4o-mini — $0.29
total) labeled 7 tasks: HelpSteer2 and SummEval ratings, MT-Bench and Arena
preferences, WMT translation scores, toxicity flags, and OracleBench math
verdicts. Prompts frozen before any simulation. Ground truth = the human
labels; strategies only "see" a label when they buy it.

Results

Pre-registered, 846 configurations × 2,000 replicates each:

• PPI++: 0.91 [0.87, 0.95] — saves ~9% of human labels. Modest, real.
• Stratified PPI++: 0.91 [0.84, 0.98] — same savings, simpler assumptions.
• Fixed PPI: 2.27× — costs labels. With mediocre judges (agreement 45–77%), untuned λ=1 adds noise. Never run PPI without power tuning.
• Disagreement routing: 2.64× — costs labels on 6 of 7 tasks, even though disagreement predicts errors (AUROC up to 0.80). Inverse-probability weights blow up variance faster than the signal helps. My prior (from Sfyraki & Wang) said this might fail. It failed.
• Judge-only: misses 18/21. Verbalized confidence is overconfident everywhere (calibration error up to 0.98).

Every default interval passed a 196-configuration coverage gate first
(Student-t intervals; stratified needs n≥100; skewed truth needs n≥200).

Limitations

Savings are ~9%, not 10× — garbage in applies. 7 English-heavy tasks, 3
mid-tier judges, one run each. Ratio estimands (F1) not shipped. Active
routing needs logged probabilities some workflows lack. High-stakes domains:
this reduces human review, never removes it.

Reproduce

pip install fewhumans
pytest  # 47 tests
python examples/toy_end_to_end.py

Repo: github.com/raihan-js/fewhumans · Dataset: JudgeBank (8 tasks, every
judge output + cost + latency) · Paper draft: arXiv soon.