There's a slide in every "LLM eval platform" pitch deck that says: score every response, catch every regression. It sounds responsible. It's also the fastest way to set a five-figure monthly bill on fire while measuring the wrong thing.
The mistake is treating "score every run" as a coverage requirement. It isn't. Coverage and sampling are two different axes, and conflating them is why teams end up paying a model to re-read 100% of their production traffic to produce a 7/10 that nobody reads.
Let me make the actual argument.
Evidence has an independence axis, not a cost axis
Before you can decide what to sample, you have to decide what kind of evidence you're collecting. This is where most eval thinking goes wrong: it ranks checks by cost (cheap regex vs. expensive GPT call) instead of by independence — how hard it is for the agent under test to forge the result.
That's the core doctrine behind agent-eval, and it splits into three tiers:
- Tier 1 — externally observable proof the agent can't forge. Did the output parse as valid JSON? Does the file it claims to have written actually exist on disk? Did the code compile? Did the tests pass? Did the run finish inside the timeout? Is the result non-empty? None of this is an opinion. The world either agrees or it doesn't.
- Tier 2 — statistical signal against a baseline the agent didn't author. Embedding similarity between the output and the original task. Length and repetition distributions. Did the diff actually change anything, or did the agent hand you back the input? The agent doesn't get to write the baseline, so it can't game the comparison.
- Tier 3 — model-as-judge. A shared-substrate opinion. Useful, but it's a signal, never a verdict.
Here's the part that reorganizes your whole pipeline: Tier 1 and Tier 2 are the real-time gate. They're deterministic, effectively free, and fast enough to sit in the hot path and block a bad run before it ships. Tier 3 is the opposite — metered, slow, non-deterministic — so it is offline-only by construction. You cannot put a model judge in the critical path without either lying about your latency or lying about your cost.
And there's a deeper reason Tier 3 stays offline: you can run Tier 1+2 over the agent's trajectory — its reasoning, its intermediate tool calls — but you cannot let a model judge that trajectory. A model grading another model's reasoning is circular: judge and judged share a substrate, so there's no independent ground truth. The judge may only inspect artifacts the judged agent didn't get to write.
Now the sampling decision writes itself
Once you accept that Tier 1+2 is the gate and Tier 3 is offline, sampling stops being about coverage. Your coverage is already 100% — every single run passes through the deterministic gate, because it costs ~$0 and runs in milliseconds. Most failures in production are boring and mechanical: stale data, a crash, malformed format, a hallucinated file path, an empty response. Tier 1+2 catches the overwhelming majority of them, alone.
So what's left for the judge? The subjective tail. "Is this summary actually faithful?" "Is the tone right?" "Did it answer the question the user meant?" That's the ~20% where deterministic checks genuinely can't reach — and it's the only place a metered model call earns its cost.
Which means the right sampling policy is not "sample 5% uniformly." It's: let Tier 1+2 gate everything, then send only the runs that passed the gate but live in the subjective tail to the judge. You're not sampling to save money on coverage you're giving up. You're sampling because you already have coverage, and the judge is for a different job.
Here's what that looks like in practice:
type Run = {
id: string;
task: string;
output: string;
trajectory: unknown[];
};
type GateResult =
| { verdict: "blocked"; tier: 1 | 2; reason: string }
| { verdict: "passed"; subjectiveRisk: number };
// Tier 1 + Tier 2: deterministic, ~$0, runs on EVERY run, in the hot path.
function gate(run: Run): GateResult {
// Tier 1 — unforgeable proof
if (run.output.trim().length === 0)
return { verdict: "blocked", tier: 1, reason: "empty output" };
try {
JSON.parse(run.output);
} catch {
return { verdict: "blocked", tier: 1, reason: "invalid JSON" };
}
// Tier 2 — statistical signal vs a baseline the agent didn't author
const sim = cosineSimilarity(embed(run.task), embed(run.output));
if (sim < 0.35)
return { verdict: "blocked", tier: 2, reason: `off-task (sim=${sim.toFixed(2)})` };
// Passed the gate. Score how likely this run needs a human-ish opinion.
// Borderline similarity == subjective territory == worth the judge.
const subjectiveRisk = sim < 0.55 ? 1 : 0.1;
return { verdict: "passed", subjectiveRisk };
}
// Tier 3 is OFFLINE. You only spend a model call on the ambiguous tail.
function shouldJudge(g: GateResult): boolean {
return g.verdict === "passed" && Math.random() < g.subjectiveRisk;
}
Notice what the judge never sees: any run that Tier 1+2 already blocked, and the vast majority of clean passes. It sees the borderline middle — labeled, explicitly, as opinion, not evidence. That single framing keeps your green dashboard honest, because you never let a 7/10 masquerade as a gate result.
The judge is only as good as the trace you feed it
There's a load-bearing assumption in all of this: that Tier 1+2 has something real to score against, and that when a run is weird you can actually reconstruct why. That's not free either. If your only artifact is the final string the agent emitted, your "trace" is a single data point and you're back to guessing.
This is why eval and observability ship as a unit, not as separate purchases. agent-eval scores and gates the output — the tiers above, drift, hallucination. AgentLens captures the trace of how the agent got there: every model step and tool call, the resolved inputs, the raw outputs, the trajectory. Two things fall out of that:
- Your Tier 1+2 checks now have unforgeable, agent-didn't-author data to score against — file writes, tool return values, resolved arguments — instead of just the agent's own summary of what it did.
- When a sampled judge call flags something, you can actually open the trace and see the step that caused it, instead of re-running and hoping the non-determinism cooperates.
Without the trace, your judge is scoring fiction and your gate is scoring the agent's marketing copy about itself.
The takeaway
Stop asking "what percentage of runs should we score?" It's the wrong question. Score 100% of them with Tier 1+2, because it's deterministic, free, and it's your actual gate. Then sample the subjective tail into an offline judge, and label its output as opinion. Ship the 80% you can prove; reserve the metered model call for the 20% you can only have an argument about.
That's the difference between an eval strategy and a very expensive way to feel covered.
