Terminal-Bench-Science: AI agents fail 70% of real research

Terminal-Bench-Science is a new benchmark that scores AI agents on real scientific research workflows, and its headline number is small: the best model resolved 30.0% of tasks. I read the announcement from the Stanford-led team behind Terminal-Bench expecting another leaderboard.

What is there is more useful than a leaderboard. It is a public, Apache-2.0 record of exactly where agents break on work that researchers actually do, plus an open door for anyone to add a task to it.

📊 A 30% ceiling is the honest headline

Here is the resolution rate across the 70 tasks in release 0.1, as published in the announcement:

Two things jump out. First, the spread between the top model and ninth place is roughly 9x, which is much wider than the spread you see on saturated coding benchmarks. Second, generation matters more than vendor: Opus 5 at 30.0% against Opus 4.8 at 10.5% is nearly a tripling within one model family.

Key takeaway: On expert-curated scientific workflows, the strongest agent available fails roughly seven tasks in ten. Treat any claim that agents "do research now" as marketing until someone shows you a number on a benchmark like this one.

🔬 The review filter is the real product

The task counts tell you more about rigour than the scores do. From the announcement:

• 920 task proposals submitted
• 464 approved for implementation
• 386 pull requests opened
• 70 tasks actually shipped in 0.1

That is a 7.6% proposal-to-shipped rate. Every surviving task passed through three separate gates:

1. A domain reviewer checking that the science is valid.
2. A technical reviewer inspecting how the task was constructed.
3. A bar raiser doing a final quality pass.

Coverage across the 70 tasks skews heavily toward three fields:

The work spans data analysis, statistical inference, simulation, optimisation, theorem proving, image reconstruction, signal processing, inverse problems, sensor calibration, model fitting, classification, and scientific machine learning. That last list is why I trust the 30% more than I trust a code-completion score: these are multi-step terminal workflows with real failure surfaces, not single-file puzzles.

🌐 376 contributors, 22 countries, and a date you can hit

This is the part I would act on. The benchmark was built by 376 contributors across 22 countries, and version 0.2 has a pull request deadline of 5 October 2026. The whole thing is Apache-2.0 on GitHub under harbor-framework/terminal-bench-science, with a Propose → Build → Review flow that starts with a task proposal form.

The announcement does not list which 22 countries, so I will not guess whether Sri Lanka is among them. What I can say is that the entry requirement is not a Stanford affiliation. It is a workflow from your own research that a domain reviewer can validate.

Look again at where coverage is thinnest:

Earth sciences: 8 tasks. Engineering sciences: 9 tasks. Between them, 17 of 70.

Now consider what gets computed here every week. Monsoon rainfall series and flood modelling. Reservoir and irrigation scheduling. Paddy and tea yield regressions against weather data. Grid load forecasting for the CEB network. Coastal erosion and bathymetry work. Sensor calibration on cheap field hardware, which is its own named category in this benchmark.

If you are a postgrad at a Sri Lankan university with a messy, real, reproducible analysis pipeline sitting in a repo, that is closer to a shippable benchmark task than most people assume. The gate is scientific validity and clean construction, both of which are effort problems rather than budget problems.

💰 What a $7,000 evaluation run tells you

The announcement publishes the cost of running the benchmark, which most leaderboards quietly omit:

Those per-task figures are my arithmetic on the published totals, not stated averages, and real cost will vary wildly by task. But the order of magnitude is the point. Around a hundred dollars of tokens buys you a 30% chance at one expert-curated scientific workflow.

If you are budgeting agent work in rupees, that reframes things:

• Do not point an autonomous agent at a long research pipeline and walk away. You are paying full price for a coin flip that lands wrong twice out of three.
• Do decompose the pipeline and use the agent on the steps you can verify cheaply and immediately.
• Do price it before you run it. Our AI agent cost calculator estimates multi-step agent runs, and the LLM benchmark comparison is worth a look before you commit to one model.

One more detail worth noticing: the announcement says Opus 5 leads in most domains, not all. In mathematical sciences, Claude Fable 5 hit 33.3% against its 21.4% overall, and GPT-5.6 Sol hit 31.4%. Picking one model for every job is leaving points on the table.

💡 What this means for you

Three practical conclusions.

If you are a student or researcher: the 30% ceiling is your job security and your opportunity at the same time. Agents cannot yet run your analysis end to end. They can absolutely draft, refactor, and check parts of it. And contributing a task to 0.2 before 5 October 2026 is a genuinely realistic line on a CV, from anywhere, for free.

If you are building on agents commercially: stop quoting SWE-bench-style numbers to clients. Terminal-Bench-Science measures something much closer to real domain work, and it says the ceiling is 30%. Scope your product to verified sub-steps, not autonomous outcomes.

If you just want to keep score: watch the gap between 0.1 and 0.2. If the top resolution rate jumps sharply on a harder, larger task set, that is a real capability signal. If it climbs while the task count stays flat, that is benchmark saturation, and you should wait for 0.3.

Bottom line: The most valuable thing here is not the ranking. It is that 70 verified failure cases are now public, licensed permissively, and open to contribution with a deadline you can plan around.