In March 2023, GPT-4 could tell you whether a number was prime with 97.6% accuracy. By June of the same year, the same model name answered those same questions correctly 2.4% of the time. Nobody pushed a bad commit. No prompt changed in your repo. The thing behind the API just... moved.
That number comes from a Stanford and Berkeley study, "How is ChatGPT's behavior changing over time?", and it's the cleanest illustration I know of the core problem with LLMs in production: the model is not a function you control. It's a dependency that can drift under your feet, silently, between two Tuesdays. And the only way you find out is if you're measuring, or if your users tell you first, which is the version where you lose.
So let's talk about how you actually measure the quality of LLM output once it's live, not in a demo. Not vibes. Not "looks good to me." A real, repeatable signal you can put a number on and gate a deploy with.
The thing that makes this hard
Traditional software is deterministic. Same input, same output, every time. You write assertEqual(add(2, 2), 4) and that test means something forever.
LLMs break that contract in two directions at once. First, the output is non-deterministic: ask the same question twice and you can get two differently-worded answers, both correct. So exact-match assertions are mostly useless; "4" and "The answer is 4." are the same answer to a human and different strings to your test runner. Second, "correct" is fuzzy. For a summarization feature there are a thousand acceptable summaries and no canonical one. You can't diff against a golden string when the golden string doesn't exist.
That's why you can't just port your unit-testing instincts over. You need evaluation that scores meaning and behavior, tolerates surface variation, and runs both before you ship and continuously after. Roughly, the job splits into three layers:
- Offline evals: a fixed test set you run on every prompt or model change, like regression tests.
- Reference-free checks: signals you can compute on a live answer with no "right answer" to compare against, like hallucination detection.
- Production monitoring: watching the stream of real traffic for drift, refusals, and quality drops over time.
Get all three and you have a safety net. Skip any one and there's a hole the bad outputs fall through.
Golden datasets: your regression baseline
A golden dataset is the LLM version of a regression test suite. It's a curated, version-controlled set of inputs paired with expected outputs (or rubrics for grading them). Every time you touch the prompt, swap the model, or bump a temperature, you run the whole set and compare the score to the last known-good run. If groundedness drops three points, you found out in CI instead of in a support ticket.
The "golden" part matters. These aren't random production samples. They're hand-curated, reviewed examples that represent the cases you actually care about, including the weird ones. The edge cases are the whole point: the empty input, the adversarial prompt, the question in the second language your product half-supports, the one customer whose data breaks your parser. A golden set of 80 sharp examples beats 8,000 random ones, because the random ones cluster around the easy middle and tell you nothing about the failures that hurt.
Here's the shape of an offline eval. Same idea in a couple of languages, because eval harnesses live in both ecosystems:
:::tabs
eval_golden.py
import json
# golden.jsonl: one {"input": ..., "expected": ..., "rubric": ...} per line
def load_golden(path):
with open(path) as f:
return [json.loads(line) for line in f]
def run_eval(model, scorer, dataset, threshold=0.85):
scores = []
for case in dataset:
output = model(case["input"])
score = scorer(output, case) # 0.0 - 1.0
scores.append(score)
avg = sum(scores) / len(scores)
# fail the build if quality regressed below the bar
assert avg >= threshold, f"quality {avg:.3f} below {threshold}"
return avg
eval_golden.ts
import { readFileSync } from "node:fs";
type Case = { input: string; expected?: string; rubric?: string };
function loadGolden(path: string): Case[] {
return readFileSync(path, "utf8")
.trim()
.split("\n")
.map((line) => JSON.parse(line) as Case);
}
async function runEval(model: Model, scorer: Scorer, dataset: Case[], threshold = 0.85) {
const scores: number[] = [];
for (const c of dataset) {
const output = await model(c.input);
scores.push(await scorer(output, c)); // 0..1
}
const avg = scores.reduce((a, b) => a + b, 0) / scores.length;
if (avg < threshold) throw new Error(`quality ${avg.toFixed(3)} below ${threshold}`);
return avg;
}
:::
The interesting question isn't the loop, it's the scorer. What does it mean to score an output from 0 to 1? That's where most of the real work is.
How do you actually score an answer?
There's a ladder of scoring methods, cheap and brittle at the bottom, expensive and flexible at the top.
Exact and fuzzy match. Works when there's one right answer: a classification label, a SQL query, a JSON field. output == expected, or a normalized version that strips whitespace and casing. Cheap, deterministic, zero ambiguity. Useless the moment the answer is a paragraph.
Heuristic and structural checks. Does the output parse as valid JSON? Does it contain the required fields? Is it under the length limit? Does it not contain the phrase "I'm sorry, I can't"? These are dumb and fast and catch a shocking amount. A spike in unparseable outputs is one of the earliest warning signs that something upstream broke.
Reference-based semantic scoring. Compare the output to a reference answer by meaning rather than string. Embedding similarity, or metrics like BERTScore. Tolerates wording differences. The catch: it needs a reference answer, and it rewards sounding similar, which isn't the same as being right.
LLM-as-a-judge. Hand the output (and optionally a rubric and a reference) to a strong model and ask it to grade. "On a scale of 1 to 5, how faithful is this answer to the provided context? Explain, then give a score." This is the workhorse of modern eval because it scales human-like judgment to thousands of cases for cents each. The MT-Bench authors found GPT-4's agreement with human raters was over 80%, roughly the rate at which two humans agree with each other. That's good enough to be useful, and just unreliable enough to bite you, which is the whole next section.
LLM-as-a-judge will lie to you in predictable ways
Here's the part nobody tells you when they sell you on LLM judges: the judge has biases, and they're systematic, which means they don't average out. They tilt your numbers in a consistent direction and you'll trust the result because it came with a confident explanation.
Position bias is the worst offender. Ask a judge "which answer is better, A or B?" and it favors whichever one came first, regardless of content. In the paper "Large Language Models are not Fair Evaluators," researchers compared Vicuna-13B against ChatGPT and found Vicuna's win rate was 51.3% when it sat in the first slot and 23.8% when it sat in the second. Same two answers, same judge; the only thing that changed was the order. Other work found GPT-4 flips its verdict around half the time when you swap positions. If your eval pipeline always puts the new prompt's output second, you've been quietly penalizing every change you ship.
The fix is cheap: run each comparison both ways and only count it as a win if the judge agrees in both orders. Ties go to "no significant difference."
judge_swap.py
def judge_pairwise(judge, question, answer_a, answer_b):
# run both orderings to cancel position bias
verdict_1 = judge(question, first=answer_a, second=answer_b)
verdict_2 = judge(question, first=answer_b, second=answer_a)
if verdict_1 == "first" and verdict_2 == "second":
return "a_wins"
if verdict_1 == "second" and verdict_2 == "first":
return "b_wins"
return "tie" # judge disagreed with itself -> not a real signal
Verbosity bias is next: judges reward longer, more elaborate answers even when a shorter one is clearer and just as correct. If your "improved" prompt produces wordier output, the judge may score it higher purely for the word count, and you measure a win that's really just inflation.
Self-enhancement bias is the sneaky one. Models tend to prefer text generated by themselves or models in their own family. Using GPT-4 to judge GPT-4 output is a little like letting a student grade their own exam. Where it matters, judge with a different model family than the one that produced the answer, or keep a human-labeled slice to calibrate against.
Warning
Don't treat a single LLM-judge score as ground truth. Calibrate it against a few hundred human-labeled examples first, measure how often the judge agrees with your humans, and only then trust it to scale. A judge you haven't calibrated is a confident number with no known relationship to reality.
Catching hallucinations when there's no answer key
Offline evals assume you know the right answer. Production doesn't give you that luxury. A user asks something you've never seen, the model answers, and you have to decide right now whether to trust it, with no reference to compare against. This is reference-free evaluation, and it's where the genuinely clever techniques live.
The strongest approach I've come across is semantic entropy, published in Nature in 2024 by a group out of Oxford. The intuition is beautiful and you can explain it in one breath: when a model actually knows something, it says the same thing several different ways. When it's confabulating, it makes up something different each time.
So you sample the same answer several times. Then, and this is the key move, you don't compare the strings, you compare the meanings. You cluster the samples by whether they entail each other (using a natural-language-inference check: do "Paris" and "It's Paris, the capital of France" mean the same thing? yes, same cluster). Then you measure entropy over the clusters instead of over the tokens.
Low semantic entropy means all the samples collapsed into one meaning: the model is consistent, probably grounded. High semantic entropy means the samples scattered across many different meanings: the model is guessing, and that's your hallucination flag. The brilliant part is it needs no ground truth at all. It catches the model contradicting itself, which it can only do when it doesn't actually know.
semantic-entropy-sketch
prompt -> sample answer 5x
"Paris" ┐
"The capital is Paris" ├─ cluster 1 (same meaning)
"It's Paris, in France" ┘
"I believe it's Lyon" ── cluster 2
"Possibly Marseille" ── cluster 3
few clusters -> low entropy -> consistent -> trust
many clusters -> high entropy -> guessing -> flag
For RAG systems there's a simpler cousin worth knowing: the RAG triad. Score three things on every answer: faithfulness (does the answer stick to the retrieved context, or invent beyond it?), answer relevance (does it actually address the question?), and context relevance (did retrieval even fetch the right documents?). The footgun the triad catches is the sneaky one: a perfectly fluent, perfectly confident answer that quietly contradicts the source document it was supposed to be grounded in. Faithfulness scoring drags that into the light.
Watching production without staring at it
Offline evals tell you a change is safe before you ship. They say nothing about the model drifting three weeks later, or a provider silently swapping the weights behind the endpoint, or your own prompt slowly rotting as five people each tweak it without writing down why. That last one even has a name now, prompt drift, the accumulated quality loss from small unrecorded edits, and it's blamed for a large share of production LLM incidents.
You don't need to grade every live request with an expensive judge. Layer it by cost:
- Cheap heuristics on 100% of traffic. Parse-failure rate, output length distribution, refusal-phrase frequency ("I'm sorry," "I cannot"), latency, empty responses. These are nearly free and a sudden spike is the earliest possible warning. A jump in refusals or malformed JSON usually means the model or the API changed under you, exactly the silent-drift case from the intro.
- Sampled LLM-judge on a slice. Run faithfulness and relevance scoring on, say, 1 to 5% of real traffic. Track the rolling average. When it dips below baseline, alert. You're not grading everything, just enough to see the trend move.
- A drift baseline. Keep a reference distribution, score histograms from a known-good period, and compare today's sample against it. When the shape shifts, something changed even if no single answer looks obviously broken.
ci-quality-gate.sh
# block the deploy if the golden-set score regressed
python eval_golden.py --dataset golden.jsonl --threshold 0.85 \
|| { echo "::error::LLM quality regression - blocking deploy"; exit 1; }
The mindset shift that makes all this click: stop treating model quality as a fixed property you verified once at launch, and start treating it as a metric you monitor like latency or error rate. It moves. Your job isn't to prove it's good. It's to notice the moment it stops being good, ideally before anyone files a ticket.
Where to start if you have none of this
You don't build all three layers in a sprint. Start with the one that pays off fastest: sit down and write twenty golden examples. Just twenty: your worst real cases, the inputs you're scared of, the ones a customer already complained about. Wire them into CI with a single LLM-judge scoring faithfulness, and gate your deploys on the average. That alone catches the prompt edit that looked harmless and broke a quarter of your answers.
Then add the cheap production heuristics, because they're nearly free and they're what would've caught that 97.6%-to-2.4% cliff on day one instead of in the news. The fancy stuff, semantic entropy, drift baselines, calibrated judge panels, comes later, when you've felt enough pain to know exactly which hole you're plugging.
The teams that sleep well aren't the ones with the smartest model. They're the ones who'd know within an hour if it got dumber overnight. Be that team.
Originally published at nazarboyko.com.
