The gap between the demo and the pipeline
In the demo, you ask the model to return a confidence score alongside its answer. It says 0.93. You write if confidence > 0.9: auto_approve() and move on.
In production, that number is just more generated text. Nobody measured it against outcomes. It looks like a probability, it's formatted like a probability, and your code treats it like a probability. That is how false precision becomes automated risk.
A confidence score earns the word "confidence" only when it is calibrated: of all the cases where the system said ~90%, roughly 90% were actually correct. Until you've checked that on your own data, the number is decoration.
Failure mode 1: the threshold that was never tested
Illustrative scenario: a team routes invoice classification through an LLM and auto-processes anything above 0.9. On their eval set, accuracy is 94%. Looks great.
What they never measured is accuracy within the >0.9 bucket. When someone finally slices it, the high-confidence bucket is barely better than the low-confidence one. The model emits 0.9+ on almost everything. The threshold wasn't filtering anything; it was a rubber stamp with a decimal point.
Failure mode 2: one accuracy number hiding two systems
If your system can defer (hand off to a human, ask for clarification, fall back to a rule), then you're not evaluating one number. You're evaluating a trade-off:
- Coverage: how often does the system answer instead of deferring?
- Selective accuracy: when it answers, how often is it right?
Push the threshold up and coverage drops while selective accuracy (hopefully) rises. Whether it actually rises is the whole question. Here's the minimal version of measuring it:
import numpy as np
def coverage_accuracy_curve(conf, correct, thresholds):
conf, correct = np.asarray(conf), np.asarray(correct)
rows = []
for t in thresholds:
answered = conf >= t
coverage = answered.mean()
sel_acc = correct[answered].mean() if answered.any() else float("nan")
rows.append((t, round(coverage, 3), round(sel_acc, 3)))
return rows
def calibration_bins(conf, correct, n_bins=10):
conf, correct = np.asarray(conf), np.asarray(correct)
edges = np.linspace(0, 1, n_bins + 1)
for lo, hi in zip(edges[:-1], edges[1:]):
m = (conf >= lo) & (conf < hi)
if m.any():
print(f"[{lo:.1f}, {hi:.1f}) n={m.sum():4d} "
f"stated={conf[m].mean():.2f} actual={correct[m].mean():.2f}")
If "stated" and "actual" drift apart in the bins you care about, your threshold is lying to you.
Failure mode 3: the global threshold
Even a calibrated score can be calibrated on average and miscalibrated everywhere that matters. Illustrative scenario: overall, the 0.9 bucket is 91% correct. Split by language and the English tickets sit at 95% while the Greek and German ones sit far lower. Split by input quality and the clean PDFs are fine while the scanned, OCR'd ones fall apart.
One threshold across all of that averages away exactly the failures you built the deferral path to catch. Run the same curve per slice:
for slice_name, idx in slices.items(): # e.g. by task type, language, source quality
print(slice_name, coverage_accuracy_curve(conf[idx], correct[idx], [0.7, 0.8, 0.9]))
If the curves differ meaningfully, you need per-slice thresholds, or you need to stop trusting the score for that slice at all.
Failure mode 4: verbalized vs. measured confidence
Asking the model "how sure are you?" and reading token log-probs are different signals, and neither is calibrated by default. Self-reported numbers in particular tend to cluster on round, reassuring values. Treat any source of confidence (verbal, log-prob, ensemble agreement, a separate verifier) as a hypothesis to test, not a fact to consume.
The reframe
Confidence isn't a model property. It's a system property you have to measure. The real question isn't "does the model feel sure?" It's "is the system's decision to proceed reliable for this task, in this context, on this kind of input?"
Practical checklist
- Log every decision with its confidence, the slice metadata, and the eventual ground truth.
- Plot coverage vs. selective accuracy, globally and per slice.
- Check calibration bins in the range where your threshold actually sits.
- Re-run it when the model, prompt, or input distribution changes. Calibration doesn't survive a model swap.
How are you deciding when your LLM pipeline is allowed to act without a human? Is anyone here actually running per-slice thresholds in production, or is everyone on a single global cutoff?
