I had a LangGraph agent running in production and a regression set of fifty queries it passed comfortably. Task completion sat at 0.93. I spot-checked the answers and they looked right. I genuinely thought I had this covered.
Then someone mentioned our cost per query had roughly doubled.
I spent an afternoon in the traces and found three things wrong at once. The router was taking the expensive path on every query that happened to mention a date. A retry loop was firing three times before the search tool gave up, and at that point the model just invented an answer. And a step near the end was overwriting a constraint that an earlier step had carefully set.
Here is the part that stung. Every single one of those still produced a correct-looking final answer. My evaluation only looked at final answers. So it saw nothing, and told me everything was fine, for a week.
That was when I realised I had been evaluating the wrong thing the entire time. Took me a while to work out what to do instead, so here is the short version.
Why the final answer hid all of it
If your agent runs in a straight line, checking the output is mostly fine. But the reason you reach for LangGraph in the first place is that your agent is not a straight line.
And everything that breaks lives in the parts a straight line does not have:
- Branches. Something reads the state and picks a path. A right answer down the wrong path is still a bug, just an expensive one.
- Retry loops. A step can loop back on itself. Five retries and a correct answer costs five times one try and a correct answer.
- Shared state. Steps read and write to a shared bag of data. One step overwriting where it should have added is silent until it is not.
- Nested graphs. A step can be a whole graph itself. Flatten it when you look and you lose where the problem was.
- Parallel work that merges. Fan out, merge back, and a collision quietly drops data.
- Memory across turns. A bug in how state gets saved might not show up until turn three.
The final answer averages over all of that. It moves when things are catastrophically broken and sits perfectly still while the agent slowly rots. Which is exactly what mine did.
The four things I score per step
The change is easy to describe and took me a while to actually do. Stop treating the agent as a conversation with an output. Start treating it as a sequence of steps, and grade the steps.
Four questions per step:
- Did it get what it needed? The right data, the right shape, nothing missing. A planning step that runs before the user's intent is filled in is broken before it starts, and everything after inherits that.
- Did it write back correctly? Did it update the shared state the way it was meant to. This is where I finally caught the overwrite-instead-of-append bug that had cost me a day.
- Did it pick the right branch? At a decision point, given the state right then, was that the correct path.
- Does it do the same thing twice? Same starting state, same behaviour after. If not, something in there is quietly random.
First two catch correctness. Third catches expensive-but-correct, which is the one that got me. Fourth catches the bugs that become flaky incidents six months later.
If you only add one check, add the branch one
This is the one I would go back and add first.
A wrong branch still produces an answer, so nothing looks broken to anyone. But a five percent drop in how often one decision point chooses correctly shows up in your cost and latency weeks before quality metrics move at all. It is a real early warning, and I had nothing like it.
It is also so much more actionable. Overall score drops a point and you are in for a two-day hunt. One decision point drops five points and you know exactly which instruction to go rewrite. On a Friday afternoon that difference matters.
Grading everything gets expensive, so I stopped
I tried scoring every step at first and the bill got silly fast. A graph of any real size does dozens of state changes per run.
What works: pick the five to ten steps whose output lots of other steps depend on. Planners, routers, the merge points after parallel work. Score those every run and sample the rest. Most of the signal, a fraction of the cost.
The checkpoint trick I had been ignoring
This is my favourite thing I learned, and it was sitting in the framework the whole time.
LangGraph can snapshot the agent's state at every step so a long job can pause and resume. That is what it is advertised for. But those snapshots are also the best debugging tool you have.
Save one at every step and every production failure becomes a test you can rerun whenever you like. Grab the snapshot ID off the failing trace, rerun from that exact point, and watch what happens. Want to check a fix? Replay the same snapshot against the old version and the new one and compare.
Before this I was guessing whether a bug would reproduce. After it I was running an actual experiment. On an incident call that is a completely different feeling.
My test set was testing my imagination
My original fifty queries were fifty things I had thought of, which means they tested my assumptions rather than my agent.
Now I pull the runs that scored badly, group the similar ones, and promote a representative snapshot from each group into the set with a note on what should have happened. Then replay all of them on every pull request.
Every incident becomes a permanent test. The set gets stronger instead of going stale, which is the opposite of what my first one did.
When this is overkill
Being honest: if your graph is five steps in a straight line with no branching, one overall score really is enough and all of this is a waste of your time.
It starts paying off once you have real branching, retries, nested graphs, or state that survives across turns. In my experience that describes most agents by about month three, whether you planned for it or not.
What I keep coming back to is that I had built a state machine and I was grading it like a chatbot. The moment I started scoring the steps, bugs that had hidden for a month were obvious in an afternoon.
If you run agents like this, I would like to hear which check caught the most for you. For me it was the branch one, and it was not close.
