What to actually measure when your agent "works"

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

The most dangerous sentence in agent development is "it works." It usually means: I ran it three times on inputs I picked, the final answers looked right, and I stopped. That's a demo result, not a measurement — and the gap between the two is exactly where agents get shipped and then quietly fail in ways nobody was watching for.

This post is about closing that gap: what to actually measure when you want to claim an agent works, and why final-answer correctness — the thing everyone measures first — is the least informative signal on the list. The short version is that an agent is a trajectory, not a function, and if you only grade the last token you've thrown away most of the evidence about whether it's reliable. Here's a five-layer scheme for what to measure instead, and a small harness that computes it.

Why final-answer correctness isn't enough

A function has an input and an output, and testing it is assert f(x) == y. An agent has an input, a sequence of decisions and tool calls, and an output. Two runs can produce the same correct final answer by completely different routes: one took 4 clean steps, the other took 22 steps, retried a failing tool nine times, burned 15× the tokens, and stumbled into the right answer by luck on the last try. Grade only the output and those two runs score identically. One of them is a time bomb.

This is the same lesson as loop drift: the agent that stays busy for 40 steps narrating confident progress can still land on a plausible final answer. Output-only grading is blind to the entire category of "right answer, broken process." And broken process is what fails you at scale, because the process is what changes when inputs get weird, the model version bumps, or a tool starts returning errors.

So the reframe is: measure the trajectory, not just the terminus. Concretely, five layers, cheapest and most obvious first.

Layer 1: Outcome — but with a real success predicate

Start with task success, because if the agent doesn't accomplish the task nothing else matters. The trap here isn't measuring outcome; it's measuring it with your eyeballs. "The answer looked right" doesn't scale past a dozen cases and it silently drifts as you get tired.

You need a success predicate: a function that takes a run and returns pass/fail without a human in the loop. For a code agent, the predicate is "does the test suite pass." For a data-extraction agent, it's "does the output match the expected schema and values." For open-ended tasks where no exact check exists, it's a rubric — and often an LLM-as-judge, which I'll come back to, because that grader has failure modes of its own. The key is that this grader is a biased instrument you need to validate, not just a black box you feed scores to.

The discipline is to write the predicate before you look at the outputs, so you're grading against a spec instead of rationalizing whatever the agent happened to produce. A predicate you tune until your current outputs pass is not measuring anything.

Layer 2: Trajectory — how it got there

This is the layer most eval setups skip, and it's the one that separates "works in the demo" from "trustworthy." For every run, record and aggregate:

• Step count — how many model→tool cycles to finish. A distribution that's creeping up run-over-run is an early warning even while the pass rate holds.
• Tool-call validity — what fraction of tool calls had well-formed, schema-valid arguments. A model fumbling a tool's schema is a tool-design problem you can see here before it becomes an outage.
• Retries and wasted steps — how many steps made no progress (repeated a call, re-derived a known fact, walked back a dead end). This is your loop-drift smoke detector.
• Terminal state — did it finish because it decided it was done, or because it hit the step cap? Cap-hits are failures even when the last answer looks fine.

None of these require a human grader. They fall out of the run log for free if you're already recording it. The reason to aggregate them is that they move before the pass rate does. Pass rate is a lagging indicator; trajectory metrics lead it.

Layer 3: Cost and latency — per task, as a distribution

Every run has a token bill and a wall-clock time, and you should treat both as first-class eval outputs, not afterthoughts. The subtlety is to look at the distribution, not the mean. Agent cost is heavy-tailed: most runs are cheap and a few pathological ones — the retry storms, the loop-drift marathons — cost 10–20× the median. The mean hides them; the p95 and max don't.

I worked the arithmetic of why retries blow up the tail in retry budgets; the eval-side takeaway is that your cost regression test should assert on a tail percentile. "Median cost held steady" can be true in the same release where your p99 doubled because one failure mode started retrying. Report p50, p95, and max cost-per-task, and alert on the tail. This matters doubly because your failures are often silent — a cost spike might be your first clue that something expensive just broke.

Layer 4: Failure class — why it failed, not just that it did

A pass rate of 82% tells you almost nothing actionable. Eighteen percent failed — from what? A wrong final answer, a schema-invalid tool call, a hit step cap, a downstream timeout, and a hallucinated tool name are five completely different bugs with five different fixes, and a single failure counter collapses them into one number you can't act on.

So classify every failure. Not with fine-grained precision — a handful of buckets is plenty to start:

• wrong_output — finished, answer failed the predicate
• invalid_tool_call — malformed or schema-violating tool arguments
• cap_hit — ran out of steps without finishing
• tool_error — a tool raised and the agent couldn't recover
• crash — unhandled exception in the harness

The value is that the shape of your failures tells you where to spend effort. If 15 of 18 failures are invalid_tool_call, you have a schema problem, not a reasoning problem, and no amount of prompt-tuning fixes it. This is the difference between a metric that scolds you ("82%") and one that points ("most failures are schema violations on the date argument").

Layer 5: Stability — pass rate is a distribution, not a number

Here's the layer that trips up people coming from deterministic testing. Run the same task twice and you can get different trajectories and different outcomes, because the model is sampling. So "does this case pass?" is not a yes/no question. It's a rate, and you only see it by running each case multiple times.

Two numbers matter, and conflating them is a classic self-deception:

• pass@k — the case passes if at least one of k runs passes. This is the optimistic number, and it's the right one only if your production system actually retries on failure.
• pass^k (all-of-k) — the case passes if every one of k runs passes. This is the number that tells you the agent is reliably right, not occasionally right.

If you run a case once, see a pass, and record "100%," you're reporting pass@1 and calling it reliability. The case that passes 6 times out of 10 and the case that passes 10 out of 10 look identical in a single run and are worlds apart in production. Measure the rate, and report the pessimistic one unless your architecture genuinely earns the optimistic one. A regression here — a case that silently dropped from 10/10 to 7/10 — is invisible to any single-run eval, and it's exactly the kind of decay that a model-version bump introduces.

A harness that computes all five

Here's a small runner that ties the layers together: it runs each case k times, records the trajectory, applies a success predicate, classifies failures, and aggregates stability and cost. It's deliberately minimal — the shape you can lift, not a framework.

from dataclasses import dataclass, field
from collections import Counter
from statistics import median
from typing import Callable

@dataclass
class RunResult:
    passed: bool
    failure_class: str | None   # None if passed
    steps: int
    invalid_tool_calls: int
    hit_cap: bool
    cost_usd: float

@dataclass
class Case:
    name: str
    task: str
    predicate: Callable[[object], bool]   # your success check, written first

def evaluate(case: Case, run_agent: Callable[[str], object], k: int = 10) -> dict:
    results: list[RunResult] = []
    for _ in range(k):
        try:
            trace = run_agent(case.task)          # returns a trajectory object
        except Exception:
            results.append(RunResult(False, "crash", 0, 0, False, 0.0))
            continue

        passed = case.predicate(trace)
        if passed:
            fclass = None
        elif trace.hit_cap:
            fclass = "cap_hit"
        elif trace.invalid_tool_calls > 0:
            fclass = "invalid_tool_call"
        elif trace.tool_errored:
            fclass = "tool_error"
        else:
            fclass = "wrong_output"

        results.append(RunResult(
            passed=passed, failure_class=fclass, steps=trace.steps,
            invalid_tool_calls=trace.invalid_tool_calls,
            hit_cap=trace.hit_cap, cost_usd=trace.cost_usd,
        ))

    passes = sum(r.passed for r in results)
    costs = sorted(r.cost_usd for r in results)
    return {
        "case": case.name,
        "pass_at_k": passes >= 1,                 # optimistic: retries save you
        "pass_all_k": passes == k,                # pessimistic: reliably right
        "pass_rate": passes / k,                  # the actual distribution
        "steps_median": median(r.steps for r in results),
        "cost_p50": costs[len(costs) // 2],
        "cost_max": costs[-1],                    # the tail is where it hurts
        "failures": Counter(r.failure_class for r in results if r.failure_class),
    }

Run that across a golden set of cases and aggregate the per-case dicts, and you get a report that answers the questions that matter: not "does it work" but how often is it reliably right, how does it fail when it doesn't, and what does the expensive tail cost me.

suite = [evaluate(c, run_agent, k=10) for c in golden_cases]

reliable   = sum(r["pass_all_k"] for r in suite) / len(suite)
recoverable = sum(r["pass_at_k"]  for r in suite) / len(suite)
worst_cost = max(r["cost_max"] for r in suite)
failure_mix = sum((r["failures"] for r in suite), Counter())

print(f"reliably right (pass^k): {reliable:.0%}")
print(f"recoverable    (pass@k): {recoverable:.0%}")
print(f"worst-case cost/task:    ${worst_cost:.2f}")
print(f"failure mix:             {failure_mix.most_common()}")

The reliable vs recoverable gap is the single most useful number this produces. A suite that's 95% recoverable but 60% reliable is telling you the agent is usually salvageable but rarely dependable — and whether that's acceptable is a product decision you can now make with a number instead of a vibe.

The grader you have to watch: LLM-as-judge

For open-ended tasks the success predicate is often another model call — "does this answer satisfy this rubric." It's the only scalable option for subjective quality, and it's also a grader with its own biases, so treat its output as a measurement that itself needs validating, not as ground truth.

The failure modes worth knowing up front: judges show position bias (favoring the first option in a pairwise comparison), verbosity bias (scoring longer answers higher regardless of quality), and self-preference (rating outputs from their own model family more generously). And a judge is happy to hand you a confident 7/10 on an answer that's fluent and wrong — the same confident-but-wrong failure that makes agents dangerous in the first place.

The cheap sanity check is to spot-audit: hand-grade a sample of what the judge scored and measure the judge's agreement with you. If your judge and a human disagree a third of the time, your "82% pass rate" has an error bar wide enough to drive a truck through. That's a whole topic — the biases, the mitigations, when to trust a model grader at all — and it's the next post I want to write. For now: never let an ungraded grader anchor a ship decision.

What I'd do

• Write the success predicate before you read the outputs. A predicate tuned until today's outputs pass measures nothing. Grade against a spec.
• Record the trajectory, not just the answer. Step count, tool-call validity, wasted steps, and terminal state are free from the run log and they lead the pass rate.
• Report cost as a distribution. p50, p95, max per task. The mean hides the retry-storm tail, and the tail is what bites.
• Classify every failure into a handful of buckets. "82%" scolds; "most failures are schema violations on one argument" points. Bucket by why.
• Run each case k times and report pass^k, not pass@1. Reliability is a rate, not a single green check. Report the pessimistic number unless your system actually retries.
• Audit your judge before you trust it. If it's an LLM-as-judge, measure its agreement with a human on a sample first. An ungraded grader is not a measurement.

"It works" is where measurement should start, not stop. An agent that produces the right answer 7 times in 10, by a route that's quietly getting longer and a tail cost that's quietly doubling, works — right up until the release where it doesn't, and then you find out you were never measuring the thing that was about to break. Measure the trajectory, the distribution, and the reason for every failure, and "works" turns from a hope into a number you can defend.