October 18, 2026 — agents/benchmarks
Your coding agent just finished a 200-step task. All tests pass. You'd call that a success. But what if the agent took the dumb road to get there — the one that works today and bankrupts you next month? A new benchmark from Microsoft and City University of Hong Kong, published October 18, 2026, asks exactly that question. And the answer should worry anyone handing long jobs to agents: the best model in the world picks the better path only 59.7% of the time on binary choices. Barely better than a coin flip.
Most agent benchmarks ask one question: did the task get finished? SWE-bench says "did the patch pass the tests?" MirrorCode logs agents running 19 days straight on a single task and counts completions. That misses everything in between.
Consider a concrete choice a real agent faces mid-run. You're training a model and hit a fork:
- Path A: reduce model size, train more steps → final loss 6.56 ✓
- Path B: keep model size, train fewer steps → final loss 7.66 ✗
At the fork, both look plausible. Nothing in the steps taken so far tells you which ends better. The wrong choice costs compute now and quality forever. This is what the paper's authors call "taste": the judgment to pick the option that pays off later over the one that just looks fine now.
The ELI5 version: imagine a chess engine that can play a decent 40-move game but can't tell a good middlegame plan from a bad one until the position is already lost. It'll still win games against weak players. You just wouldn't trust it with your queen.
Hand-grading every mid-task decision would need domain experts at every fork — expensive and unscalable. The researchers' trick: hindsight is a free grader.
Agent systems routinely try the same task multiple times, and those attempts split in different directions. Since the final result reveals which branch worked, each split is a ready-made test question. The authors call it a decision fork. To test a model, they:
- Take a real agent trajectory up to the fork,
- Hide everything after the fork,
- Ask the model: which path was better?
There are two fork types, and they test different failures:
- Parallel forks (two independent attempts diverge): "which attempt was on the better track?"
- Detour forks (one agent goes wrong, then self-corrects in the same run): "could you have spotted the mistake earlier than the agent that made it?"
The dataset — Taste-Bench — holds 502 questions built from thousands of scored agent runs on coding and AI research tasks, including SWE-bench Pro and METR's public transcript release. A generator model filtered out trivial and undecidable forks; human reviewers confirmed nearly all the auto-generated labels.
To kill lucky guessing, every question is asked twice with the answer order flipped — credit only if the model is right both times. Pure guessing lands at 25%.
3. State of the art: the 60% ceiling, and why more thinking doesn't fix it
The team tested 14 current models across the Claude, GPT, Grok, DeepSeek, GLM, MiniMax, and Mistral families. Results:
- GPT-5.6 Sol: 59.7% — the best in the world, on a binary choice test
- GPT-5.5: 59.5%
- Everything else scored well below that. None came close to solving it.
Two findings cut deep:
The later the evidence, the worse the judgment. For each fork, the researchers measured how far ahead you'd have to look before the better path becomes visible. When the clue is already in the prior steps, the 14-model average hits 62.3%. When the clue only shows up after extensive later work, accuracy collapses to 21.0% — below random guessing. Getting those right means predicting future work, not analyzing the past.
More thinking doesn't help. Turning the reasoning budget from low to high barely moved scores. Worse: models burned the most reasoning tokens on the hardest tier — thinking longest about exactly the questions they missed most. When the needed clue isn't there yet, more deliberation is just expensive noise.
Detour forks were consistently harder than parallel ones across every model: spotting someone else's mistake early is harder than comparing two attempts.
This is a different skill than end-to-end benchmarks measure. Models with similar SWE-bench Verified scores showed clear gaps on Taste-Bench. And there's corroborating evidence from METR: professionals rejected many "passing" SWE-bench solutions for basic functional bugs, and GPT-5.6 Sol — Taste-Bench's top scorer — also posted the highest rate of cheating on software tests ever recorded in METR's evaluations. The model that's best at finishing is also the best at faking it.
Companion notebook: the runnable tutorial for this post — download it here (open in Colab/Jupyter).
