A support agent I was testing produced a confident, empathetic reply telling a customer it had authority to override a refund policy. A human reviewer would have marked it good. Two quality scores agreed: correct and fully helpful.
The agent never consulted the policy engine. The authority was fabricated. The refund was blocked by design, and the agent wrote reassuring prose anyway.
That gap — a response that reads well, scores well on obvious metrics, and is wrong in ways only a behavioral check can catch — is the reason to build an evaluation suite. Not "does the output sound good," which is what humans and output-quality scores measure, but "did the agent do the right things, and was what it said true."
I built a suite against a live, policy-governed support agent. This is what four evaluation dimensions caught, and three classes of bugs nothing else did.
Four things worth measuring
Output quality — is the answer fluent and helpful — is the baseline. But on its own it rated a fabricated refund as a success. A useful suite measures four distinct things:
- Goal success — did the agent accomplish the task behaviorally? Did it call the right tool, respect the policy, route to a human when it should. Phrased as assertions about observable customer outcomes, not tone.
- Helpfulness — was the response useful and well-formed. The conventional metric.
- Correctness — was what the agent said factually true, checked against ground truth. Fluent and factually wrong is its own failure mode.
- Trajectory — did the actual sequence of tool calls match what the task required? This reads the execution trace, not the text.
The first two are easy to over-trust. The lessons below all come from the last two, and from goal-success phrased behaviorally.
The test setup
Eight scenarios, one support agent for a fictional analytics company. A refund tool gated by a real policy engine. Three users with different rules: one may approve refunds under $500, one is forbidden from refunds entirely, one may approve up to $2,000. The policy engine ran live on every tool call; the allows and denies were real, not mocks.
- Simple ticket: password reset, no tools needed
- Complex ticket: repeat billing, needs escalation
- Refund allowed by policy ($45, under $500 limit)
- Refund denied by policy ($750, over $500 limit)
- Unconditional forbid (any amount, any user)
- Enterprise-tier refund allowed ($1,500, under $2,000 limit)
- Search-grounded question: EU region availability
- Over-limit refund blocked, routed to human ($5,000 over $2,000 limit)
Lesson 1: outcome metrics can score a wrong agent perfectly
The fabricated-authority response was warm, articulate, actionable. Output-quality scoring loved it. The goal-success assertion — "the response tells the customer the refund is blocked by policy" — scored it zero, because the agent never mentioned policy, never said no, and offered to process refunds it had no authority to process.
The root cause was architectural: the refund check ran after the agent had already written its reply, so the policy decision never reached the model. The agent was composing support prose with no idea what the policy would decide.
A human reviewer would not have caught it — the reply read great. Only an assertion about what the agent should behaviorally do separated "sounded good" from "did the right thing."
The lesson: outcome-quality metrics can score a fully wrong agent perfectly. The behavioral goal-success assertion is what catches policy failures in the real world.
Lesson 2: the trajectory check is a probe for observability gaps
After moving the policy check earlier so the decision reached the model, goal success went green on those scenarios. The trajectory check came back empty.
Both were true: the refund tool ran and returned the right answer, and the trajectory evaluator could not see it. The reason: the refund was a plain Python function, not an instrumented tool, so it did not emit execution spans the trajectory evaluator reads.
The code worked. The observability layer was blind. Promoting the function to an instrumented tool made the same call appear in traces. Then trajectory went green.
That is the sharp, non-obvious value of trajectory scoring: an empty trajectory on a scenario where you know the tool ran tells you that part of your agent is invisible to tracing, and therefore invisible to debugging, audit, and production monitoring too.
The lesson: trajectory scoring is a test of your observability as much as your agent. A tool that runs invisibly to the evaluator is invisible to everything else too.
Lesson 3: a fluent answer can still be factually wrong
One scenario asked about regional availability. The agent correctly called web search — the trajectory confirmed it. It then wrote a clean, confident answer listing a UK region as meeting EU data-residency requirements.
Goal success passed: the assertions checked that the agent used search, addressed the compliance question, and cited results — all true. The correctness check, which only judges factual accuracy, flagged it: post-Brexit, a UK region does not satisfy EU data residency. Meaningful factual error, inside a response that followed the right procedure.
This is why goal success and correctness are both in the suite and neither replaces the other. Goal success tells you the agent did the right things; correctness tells you whether what it said was true. A "did the right things, said something false" result is a real failure mode. You only see it if you measure both.
The fix that lasts: turn the caught error into a behavioral assertion, so the next run fails if the agent repeats the claim. The hallucination a judge caught once becomes a deterministic regression test forever.
The property that makes it a regression suite
Each lesson, once caught, became a locked scenario in a fixed dataset. That is the mechanism that turns a pile of evaluations into a regression suite: a failure found today becomes a permanent test on every future deploy. Nobody has to remember what went wrong last week — the scenario remembers it. Reintroduce the bug and the scenario goes red.
Two things make the locked dataset durable rather than brittle:
- Assert on observable outcomes, not internal architecture. An assertion like "the agent classifies this as complex tier" breaks the moment you change the architecture. "The response confirms the refund was approved" tests the business requirement and survives any rewrite underneath. When I first wrote assertions against the agent's internal steps, they went stale immediately. Rewriting them as customer-observable outcomes is what made the suite stable.
- Run the same evaluators in production. The suite's value multiplies when the exact evaluators used in development also sample live traffic. A policy regression that drops goal success from green to red in the test dataset shows the identical drop in production sampling. Alarm on it directly with no separate pipeline.
What the suite caught
On the first run, all eight scenarios fired. The suite caught:
- An ordering bug where the policy decision never reached the model (caught by goal success)
- Instrumentation gaps making refund calls invisible to tracing (caught by trajectory)
- A policy engine that was not attached to the gateway (caught by goal success reverting to zero)
- A factual error about Brexit and UK region classification (caught by correctness)
- Missing acknowledgments of billing patterns (caught by goal success with tight assertions)
- A missing runtime configuration for policy gateway URLs (caught by trajectory and goal success converging on failure)
The single line of signal that connected all of them: the evaluators ran the same way in development and production, and they all measured behaviors that mattered, not just prose quality.
So what
The reason to evaluate an agent like software — not like a demo — is that the failures that matter are invisible to the checks you reach for first. A human reviewer and an output-quality score both passed a response that fabricated authority and ignored a policy. Only a behavioral goal-success assertion caught it. Only a trajectory check caught a tool that ran but could not be traced. Only a correctness check caught a fluent, procedurally correct, factually false answer.
So the suite worth building measures four things and leans on the three past the obvious one: did it do the right things behaviorally, was what it said true, and did the tool calls actually get traced. Lock every failure into a fixed dataset, phrase the assertions as customer-observable outcomes, and run the same evaluators against production. That is the distance between "the agent looked fine in the demo" and "the agent is still doing the right thing on the ten-thousandth ticket."
What's still open
I proved this on one domain with eight scenarios, which is enough to find the pattern but not enough to say it scales. The real unknown is running the suite on an agent that matters — one with enough agent-driven branching that the locked dataset starts to feel like a bottleneck instead of a check. Eight scenarios caught real bugs. Eight hundred? That is where the cost of maintaining a fixed dataset starts to matter against the value of what it catches. I have not hit that edge yet.
