The Hard Part of Agent Evals Is Deciding What Counts as Green

AI agent products still need conventional E2E tests. The UI should open, APIs should behave as expected, permissions should hold, and persistence should not break.

If deterministic software breaks, I want a deterministic test to turn red.

But when you build an agent product, you increasingly want another lane next to that: an eval lane that runs the real model and runtime and evaluates the agent itself.

This is no longer a niche concern. Anthropic describes automated evals as especially useful for pre-launch validation and CI/CD, and notes that Claude Code has added evals for things like concision, file edits, and over-engineering. The same article describes Descript separating quality benchmarking from regression testing, and Bolt combining static analysis, a browser agent, and LLM judges.

But once you start building this kind of system, the hard part is not running the eval.

It is deciding what counts as Green.

I have been working through this recently while building a real-model agent eval lane. I do not think there is a universal answer yet for what should become a production gate.

The question I keep coming back to is:

Which evaluations have earned the right to block a release?

Agent evals have too many things we could measure

A traditional LLM eval has a relatively compact mental model:

prompt
  ↓
model / workflow
  ↓
response
  ↓
grader

Correctness, relevance, groundedness, formatting, LLM-as-a-Judge. Of course evals have long been more sophisticated than this, but agents expand the evaluation surface dramatically.

Once a model can use tools, modify state, retain information, and operate across multiple steps, performance is no longer determined by the model alone. The environment and agent harness matter too.

Now we can evaluate:

final response
tool selection
tool arguments
trajectory
task completion
state mutation
artifact quality
safety
latency
cost
recovery

All of these can be useful. That is exactly why the problem gets harder.

Agent evals increasingly suffer less from a lack of measurable things than from an excess of them.

A production gate does not need more metrics. It needs a much smaller question:

What failure are we unwilling to ship?

Decide the claim before the score

Suppose an agent eval has a 72% pass rate. Is 72% good?

The number alone cannot tell you. If this is a new capability and the same eval was at 40% last month, 72% may be excellent. If this is a regression suite protecting behavior users already depend on, 72% may be unacceptable.

Maybe some failures came from the environment. Maybe the grader rejected valid solutions.

Anthropic makes a useful distinction between capability evals and regression evals. Capability evals measure what an agent can currently do and may reasonably begin with a low pass rate, while regression evals protect behavior that is already expected to work and should usually be much closer to 100%.

So I have stopped starting with the threshold. I prefer this order:

product claim
     ↓
   oracle
     ↓
   metric
     ↓
 threshold

"95% means Green" is not a meaningful production gate unless the 95% is attached to a clearly defined claim.

The claim and the oracle are what give the threshold meaning.

Look at the world before looking at the model's words

In the eval lane I have been building, I use one rule almost mechanically:

If the environment already knows the answer, do not use the model's prose as the oracle.

If the agent says it created an artifact, check whether the artifact actually exists and whether it is valid. If something was supposed to remain unchanged, compare the state before and after.

If the agent must stay inside a boundary, observe the actual side effects. If an operation should be reversible, roll it back and inspect what remains.

The response text itself does not need to be the center of the pass/fail decision. For many agent tasks, the environment knows more than the transcript does.

Anthropic describes a similar distinction between transcript and outcome. An agent saying "I booked the flight" is a fact about the transcript; whether the reservation actually exists is a fact about the world.

$\tau$-bench takes a similar approach for tool-using agents by comparing the final database state with an annotated goal state.

Once you start evaluating agents this way, something interesting happens:

a large part of the eval starts looking like ordinary software verification.

But final state is not always enough.

I ran into this earlier while fuzzing schema migrations. The final schema looked correct, so the test was Green.

But the path was effectively:

DROP TABLE
CREATE TABLE

The destination was correct. Production data would still be gone.

The lesson was:

destination correctness and path safety are different claims.

The same applies to agents. Some properties can be judged entirely from final state, while others cannot.

If the contract requires:

authorization
    ↓
mutation

then the ordering itself matters, and you need to inspect the path.

So my current rule is:

Constrain the path only when the path itself has safety or policy meaning. Otherwise, prefer the outcome as the oracle.

Agents may discover valid paths we did not anticipate. That is usually a feature, not a bug.

Use the cheapest honest judge

Thinking in terms of product claims made it easier for me to separate judges by what they can actually answer.

Product claim	Judge
Deterministic product machinery works	unit / integration / E2E
Agent reaches a verifiable state	deterministic outcome check
Agent never crosses a hard boundary	deterministic invariant
Agent follows a required policy path	trajectory / tool constraint
Output is useful, natural, or elegant	calibrated LLM / human judge

I think of this as the cheapest honest judge.

Cheaper is better, and faster is better, but only if that layer can actually answer the property. There is no reason to ask an LLM whether a file exists.

At the same time, "Was this response genuinely useful for the user's intent?" may not be something a simple boolean function can answer.

Use semantic judges for semantic properties. Use facts where facts exist.

The agent can be stochastic. The judge does not have to be.

A real model makes the execution stochastic. The same task may produce different wording, tool ordering may change, and multiple solution paths may be valid.

That makes a single Green/Red run a weak description of quality. In the lane I am building, I run the real model repeatedly and measure success across runs rather than treating one execution as definitive.

But the component that maps observed facts to pass/fail does not also need to be stochastic.

A useful separation is:

real agent
    ↓
stochastic execution
    ↓
observable facts
    ↓
small deterministic judge

That deterministic judge can itself be exercised extensively with ordinary unit tests. In my setup, contract-violation logic is separated from the real-agent E2E path so I can test it without the model, network, or runtime.

There is a slightly paradoxical lesson here:

To make agent evals more reliable, I try to remove the agent from as much of the eval system as possible.

Use the real model where only the real model can produce the behavior you need to observe. Then get back to ordinary software as quickly as possible.

Not every run should be Green or Red

Real-agent evals have another problem: the environment fails too.

Rate limits, connection failures, remote service failures. Anthropic recently reported running Terminal-Bench 2.0 under different infrastructure configurations and seeing a six-percentage-point gap between the most and least resourced settings. In some configurations, infrastructure errors unrelated to model capability were themselves around 6%.

If every one of those runs becomes a product failure, your pass rate stops meaning what you think it means.

I have found three verdicts more useful:

pass
fail
inconclusive

inconclusive means:

this run did not produce enough evidence to evaluate the product claim.

But that does not mean all evidence should be discarded.

Suppose the model connection dies halfway through a turn, but before that happens the agent has already crossed a boundary it was never allowed to cross. Task completion is inconclusive; the boundary violation is not.

An environment failure should not erase an independently observed contract violation.

The rule I take from this is:

"The run did not finish" and "nothing bad happened" are not the same statement.

Run status and observed invariant violations should be tracked separately.

If you want to trust Green, break the judge first

There is one more uncomfortable part of evals: the grader itself can be wrong.

OpenAI audited SWE-Bench Pro in July 2026 and estimated that roughly 30% of the public tasks had breaking issues. Problems included tests enforcing implementation details that were not required by the prompt, hidden underspecified requirements, and weak tests that allowed incomplete fixes to pass.

A sophisticated agent, a sophisticated harness, and a large eval suite can still fail in a much simpler place:

a bad oracle can measure the wrong thing very precisely.

So I want to test the judge itself. Feed it known bad states and make sure they actually turn Red.

known bad state
      ↓
    judge
      ↓
 does it fail?

A state that should have remained unchanged was mutated. A forbidden side effect occurred. A result that should exist once was duplicated.

If the judge cannot reliably catch violations I already know about, I have little reason to trust its Green results.

The judge is part of the system under test too.

If you want to trust Green, deliberately break something first.

Not every eval deserves to block a release

None of this means every agent eval should go into blocking CI. Real-model runs cost money, they are slower, they depend on remote infrastructure, and they are stochastic by design.

More importantly, evals have different jobs. Some explore capability, some calibrate new behavior, some compare quality, and some protect an existing product contract against regression.

Those should not all share the same threshold or release policy.

The real-agent eval lane I am building today is still separate from conventional deterministic E2E and is not itself a blocking release gate. It runs the real model repeatedly and accumulates evidence.

The question is not how many evals we can add.

It is:

Which evals are mature enough to justify saying, "If this regresses, do not ship"?

For me, an eval probably needs something like:

clear product claim
    ↓
explainable oracle
    ↓
grader itself is tested
    ↓
environment noise is separated
    ↓
stability is measured across repeated runs

Simply having an eval score is not enough.