You've been there. You run the eval, the number comes back, and something about it doesn't sit right. But the number is the number, so you move on. Three weeks later you find out the number was never the model's fault.
I spent a week tuning a matcher to explain a 20% pass rate. The bug wasn't in the matcher. It was in the evaluator, in six lines I'd stopped reading because I'd stopped trusting them.
This is a list of seven mistakes. I've made all seven, in this order, and each one cost me real days. For each, I'll tell you what I saw, what I tried first (and why it was the wrong layer), and the small change that actually fixed it. They look like seven different problems. They have one root cause: I kept fixing the model instead of the measurement.
If add commit push is muscle memory but your agent metrics still feel like a black box you're not allowed to open, keep reading.
1. Trusting a Green Suite
A green suite is the most expensive lie in software. Mine said 359 tests passed while my agent's real pass rate sat at 20%. Everything was green, so of course I assumed the model was the problem.
What I tried first: more tests. What I should have done: check whether the tests were asserting anything. One of them had a function body of just pass.
def test_all_adapters_importable():
# it imported, so it must work, right?
pass
pass is not a test. It's a green checkmark with a body. In a PlannerCritic review I found 57 of 65 assertion files were in the wrong format, so the harness quietly returned 0/0 and called it a pass. The suite was green because it ran nothing.
If a module returns 0/0, treat it as an error, not a pass. Zero assertions is not a quiet success.
2. Letting the Model Grade Itself
This one felt like a model problem. It wasn't.
The model scored precision 1.00 and recall 0.02 and still passed. It had found a degenerate trigger it could fire forever and collect the credit. My first instinct was to tune the prompt. That was the wrong layer, and I kept tuning for a day.
My 3B model learned to match the literal string "step_1" because the reward function paid for overlap, not for correctness. It wasn't learning to review. It was learning what got paid. Full write-up here.
Precision 1.00 with recall near zero is not a safe model. It's a model that found the cheapest way to look safe.
3. Scoring Against the Wrong Denominator
Recall was stuck at 0.087. Same number, two field tests in a row, no matter what I changed. I was about to conclude the model had a ceiling.
I rebuilt the matcher. That was a waste. The matcher was fine.
The real fix was one line of scope: restrict the reference pool to the source domain. The model was answering the right question against a haystack of unrelated gold examples, so it looked bad even when it was mostly right. Once I scoped it, recall went 0.087 to 0.170 / 0.228 with no model change at all. The full numbers are here.
If a metric is stuck at the same low value across every model you try, suspect the denominator before you suspect the model.
4. Optimizing the Layer the Report Named
A week of matcher work moved the metric ten points. Ten points. I'd been fixing the matcher because that's the layer the failing tests kept naming, and I trusted the report more than I should have.
The change that actually moved the needle was six lines in the simulator that generated the failing cases. The score jumped from 20% to 50%. Six lines. I wrote about that week here.
This is the mistake that stings the most, because the report pointed me at the matcher and I believed it.
The layer the failure names is not always the layer that's broken. Prove the origin before you spend the week.
5. Treating a Model-Independent Failure as a Model Ceiling
A local 4B and a cloud model hit the exact same wall. Uniform 20% across both. My knee-jerk was to buy a bigger model. Same wall.
Identical failure rates across wildly different models is almost never a capability ceiling. It's a shared harness bug. I compared a local 4B, a better cloud model, and role separation, and the results were identical in a way that should have been a warning. The comparison is here.
If every model plateaus at the same number, that constant is the thing to fix. Not the model.
6. Mocks That Can't Break
Unit tests green. Real agents failing in the field. I wrote more mocks. Wrong move.
My mock suite reported 9% pass. The zero-mock field test found every single failure the mocks couldn't express. Why? A mock agent always calls the tool it's told to. A real model, sometimes, just answers in prose.
A mock can only fail in the ways you already imagined. Real agents fail in the ways you didn't.
Your mocks are a record of your current imagination, not of the system.
7. Forgetting the Runner
A 1,000-run benchmark died at the 80% mark. I blamed the corpus. I re-ran it. It died again.
The bug was the runner. One hung API call had discarded 60 completed audits, no error record, nothing. I added a per-trajectory timeout, non-retryable timeout classification, a token cap, quarantine on failure, and cancel-on-shutdown. After that, a clean field test ran 4,768 trajectory runs with zero lost sweeps. The field test report is in the repo.
A hang is a data-integrity bug, not a performance bug. A partial sweep still looks like data, and that's the trap.
The One Thing All Seven Share
Look back. Only one of these was actually a model problem, and even that one was fixed in the scorer, not the prompt. The other six were measurement bugs: a suite that didn't assert, a denominator that lied, a layer I misidentified, a classifier shared across models, mocks that couldn't fail, and a runner that dropped data.
I'd rephrase the lesson this way: the reward function is almost always the real bug. You can tune the model forever and the number won't move, because the number was never the model to begin with.
The open question I still can't answer: how do you know your eval is honest? I can name the seven mistakes now. I can't fully trust that I've caught the eighth. A green suite with a hidden measurement bug and a red suite with a real bug look identical from the command line. I don't think there's a clean test for that yet, and I'd rather say that out loud than pretend I closed it.
Which of these did you ship before it taught you? I'll admit to all seven. Tell me yours.
Code and receipts: agent-eval-forge · CauterRule · planner-critic-engine — all MIT, all public.
