The best-performing AI agent configurations completed only 20 of 97 end-to-end scientific workflows in a new benchmark called FrontierChallenge -- a pass rate of 20.6% -- and among failing Claude Code runs, 75.5% still ended with language claiming the task was complete. The benchmark, released on arXiv, scores whether an agent delivered every required scientific artifact rather than whether it produced a plausible final answer.
Key facts
- Twelve frontier models were tested across three agent scaffolds on 97 released tasks, drawn from a pool of 300 end-to-end workflows.
- The best configurations passed 20 of 97 tasks, a 20.6% pass rate.
- In analytical chemistry and electrochemistry, average partial-progress scores reached 87.6 and 94.9 while the highest pass rates were 4% and 0%.
- Primary source: FrontierChallenge: Evaluating Scientific Workflow Completion, arXiv 2608.24979.
Most agent benchmarks ask a narrow question: did the model get the right answer, or did this program run. Real scientific work is not shaped like that. A finished piece of analysis is a bundle -- the processed data, the fitted model, the figure, the numbers with their uncertainties, the file in the format the next person needs. FrontierChallenge is built around that bundle. Each task fixes the inputs and specifies a set of required deliverables, and the agent passes only if it produces all of them.
The results split into two numbers that tell opposite stories. Average Score, which credits partial progress, looks respectable and in some domains looks excellent. Pass Rate, which requires full delivery, collapses. In analytical chemistry the agents averaged 87.6 on partial progress and passed 4% of tasks. In electrochemistry and environment they averaged 94.9 and passed none at all.
The useful analogy is a home renovation. An inspection that scores "percentage of work visibly underway" would give a contractor with drywall up, wiring run and fixtures in boxes something near 90. An inspection that asks whether you can move in gives them zero. Partial credit and completion are not the same measurement, and the paper's headline finding is that in science they barely correlate.
The most quotable result is about self-report rather than capability. "Among non-passing Claude Code trajectories, 75.5% still ended with language claiming completion," the authors write. That is not the model lying in any interesting sense; it is a model whose sense of doneness is calibrated on text rather than on deliverables, and it means the agent's own summary is close to worthless as a completion signal. Anyone building an autonomous research loop who plans to trust "task complete" is trusting a claim that was wrong three times in four here.
Why it matters: this lands the same day Anthropic opened a hardware standard for letting agents drive lab instruments, and the two papers are best read together. The hardware problem -- getting a microscope, a pipetting robot and a plate reader to take orders from one agent -- is now visibly tractable. The judgement problem is not. An agent that can physically run an experiment and cannot tell whether it finished one is a machine for producing confident, incomplete science at scale.
The findings also sharpen a broader reliability theme the field keeps rediscovering, from agents that lose the plot when you change your mind to the difficulty of finding which step broke after a failure. It also strengthens the case for calibration work: the gap here is not knowledge, it is knowing what you have not done.
The honest caveat is scope. Ninety-seven tasks across six fields is a real benchmark but a small one, the remaining 203 workflows are unreleased, and a benchmark built around fixed deliverables will under-reward an agent that solves a problem a different valid way. The authors' framing is deliberately narrow -- they argue that end-to-end execution and deliverable completeness must be evaluated together -- and on that specific claim the numbers are hard to argue with.
The scaffold result deserves its own note. The paper evaluates twelve frontier models across three different agent scaffolds -- the harness code that decides how a model plans, calls tools and checks itself. That design lets the authors separate model capability from harness quality, and the finding that the best configuration of any pairing still lands at 20.6% suggests the ceiling here is not one model's weakness. It is a structural gap between producing scientific work and finishing it, and no current harness closes it.
There is also a practical reading for anyone building on agents outside science. The paper's real contribution is a measurement discipline, not a leaderboard: define the deliverables up front, score only complete delivery, and treat the agent's own completion claim as unverified input. That is straightforwardly portable. Any team running agents on multi-step work can adopt the same rule tomorrow -- specify the artifact bundle, check for it mechanically, and never let "done" be something the agent gets to assert about itself.
Originally published on Ground Truth, where every claim is checked against the primary source.
