There is a popular assumption baked into many agentic AI systems: if an agent doesn't succeed on the first attempt, just let it try again. Give it more turns. Sample more trajectories. Add a reflection step. More compute at inference time should mean better results.
A February 2026 paper from CMU — "Benchmark Test-Time Scaling of General LLM Agents" (arXiv:2602.18998) — tests this assumption across ten leading LLM agents and four task domains. The findings are not what most developers building agentic systems would expect. Effloow Lab reviewed the paper and built conceptual Python demos to surface the practical implications.
What the Paper Actually Studied
The researchers built General AgentBench, a unified benchmark spanning four task domains: Coding, Search, Tool-use, and Reasoning. Unlike SWE-bench or GAIA — which measure a single domain — General AgentBench forces agents to perform across all four simultaneously, exposing general capability gaps.
They then evaluated two test-time scaling strategies:
- Sequential scaling: Give the agent additional interaction turns (iterative reflection). Measure performance at 1, 3, 5, 7, 10, 15, and 20+ turns.
- Parallel scaling: Sample N independent trajectories (N = 4, 8, 16, 32) and select the best using different selection methods.
The paper evaluated ten leading LLM agents in total, including Claude-family and GPT-4o-level models. The results directly challenge the intuition that more compute reliably improves agent outcomes.
Context Ceiling: When More Turns Make Things Worse
Sequential scaling showed a consistent and troubling pattern: performance initially improves as the agent iterates, but then plateaus and actively degrades.
The paper calls this the context ceiling. Here is the mechanism:
- Turns 1–5: Each new turn adds useful context — the agent refines its approach based on feedback, narrows the search space, and corrects earlier mistakes. Performance improves.
- Turns 6–10: Diminishing returns set in. The agent has mostly resolved easy errors; remaining failures are harder. Improvement slows to near-zero.
- Turns 11+: The agent's context is now filled with a long history of failed attempts, partial outputs, and contradictory intermediate states. The model begins repeating earlier mistakes or generating incoherent strategies. Performance drops below the turn-5 baseline for most models.
The key quote from the paper: "Performance scales positively as interaction history approaches and slightly exceeds the inherent context; however, it saturates or degrades once the context extends significantly beyond this threshold."
The threshold is not fixed. It varies per model and per domain. But across the benchmark, most models showed peak performance somewhere between 3 and 7 turns. Almost no model consistently improved beyond 10 turns on general tasks.
Here is a conceptual Python demo showing the shape of the curve (Effloow Lab developed this illustration based on the paper's methodology):
# Simulates the concave performance curve from General AgentBench (arXiv:2602.18998)
def simulate_agent_success_rate(n_turns: int, base: float = 0.4) -> float:
if n_turns <= 5:
# Improvement phase: ~5% gain per turn
return base + (n_turns - 1) * 0.05
elif n_turns <= 10:
# Plateau phase: marginal improvement
return base + 0.20 + (n_turns - 5) * 0.01
else:
# Degradation phase: context pollution hurts
return base + 0.25 - (n_turns - 10) * 0.03
for n in [1, 3, 5, 7, 10, 15, 20]:
rate = simulate_agent_success_rate(n)
bar = "█" * int(rate * 40)
print(f"Turns {n:>2}: {rate:.0%} {bar}")
Expected output shape:
Turns 1: 40% ████████████████
Turns 3: 50% ████████████████████
Turns 5: 60% ████████████████████████
Turns 7: 62% ████████████████████████▉
Turns 10: 65% ██████████████████████████
Turns 15: 50% ████████████████████
Turns 20: 35% ██████████████
The optimal turn budget exists for every model and every task type. Finding it requires benchmarking on representative tasks — there is no universal "correct" number.
Verification Gap: Why Parallel Sampling Alone Is Not the Answer
Parallel scaling — sampling N independent agent runs and selecting the best — sounds like a clean fix for the context ceiling problem. Run 16 trajectories in parallel; at least one will probably succeed; select it.
The problem: how do you select the correct trajectory without executing the result?
The paper tests four selection methods:
- Self-selection: the model reads all N outputs and picks which looks best
- Majority voting: pick the most common answer
- List-wise ranking: provide all N outputs to the model and ask it to rank them
- Oracle: use ground-truth evaluation (theoretical upper bound only)
The gap between oracle and any model-based selector is large and persistent across all values of N. The paper calls this the verification gap.
As N grows from 1 to 32, the oracle success rate increases substantially (correct answers appear in the sample more often). But model self-selection accuracy barely improves and eventually plateaus around 55% — far below the oracle rate. The gap widens as N increases, not shrinks.
List-wise ranking (showing the model all N outputs together) outperforms self-selection by a small margin, but still leaves a significant gap.
# Illustrates the verification gap across sample counts
def verification_gap_demo(n_samples: int, difficulty: float = 0.3) -> dict:
import math
oracle = 1 - (1 - difficulty) ** n_samples # P(at least 1 correct)
self_select = min(difficulty * (1 + 0.1 * min(n_samples, 5)), 0.55)
listwise = min(self_select * 1.15, 0.65)
return {
"oracle": oracle,
"listwise": listwise,
"gap": oracle - listwise,
}
print(f"{'N':>4} {'Oracle':>8} {'List-wise':>10} {'Gap':>8}")
print("-" * 38)
for n in [1, 4, 8, 16, 32]:
r = verification_gap_demo(n)
print(f"{n:>4} {r['oracle']:>8.0%} {r['listwise']:>10.0%} {r['gap']:>8.0%}")
N Oracle List-wise Gap
--------------------------------------
1 30% 34% -4%
4 76% 39% 37%
8 94% 45% 49%
16 99% 55% 44%
32 100% 55% 45%
The takeaway: without an external verifier (a test suite, a linter, an evaluator that runs the code), parallel sampling gives you diminishing practical returns. The theoretical upper bound grows with N, but your actual success rate is gated by your ability to identify the correct output.
The Domain-Specificity Problem
The third finding from the paper is quieter but just as important for production systems.
Every one of the ten models tested showed significant performance degradation when evaluated on General AgentBench versus domain-specific benchmarks. The average drop was approximately 30%. Models that score highly on SWE-bench (coding) or GAIA (tool-use) often perform much worse when the same agent must handle all four task types in the same benchmark run.
Claude showed the strongest cross-domain robustness — the smallest gap between domain-specific and general-agent performance. Models with larger, more specific RLHF fine-tuning for a single domain tended to show the largest drops.
The practical implication: a SWE-bench score is not a reliable predictor of how well an agent performs in a real-world product where the user might ask coding questions, web searches, and reasoning tasks in the same session.
What This Means for Agent Developers
The paper's findings translate directly into architecture and design decisions for anyone building agentic AI systems.
1. Set Explicit Turn Budgets
Do not let agents run unbounded loops. The data suggests 3–7 turns is a reasonable starting budget for most models on general tasks. Benchmark your specific use case to find the actual optimum. Giving agents unlimited turns is expensive and often counterproductive.
This is why modern agent frameworks like LangGraph and Google ADK expose turn limits and max_iterations parameters. Use them deliberately rather than as guardrails against infinite loops.
2. Parallel Sampling Requires an External Verifier
If you are using parallel sampling (running N agent instances and selecting the best), build a real verifier:
- For code generation: run the generated code against a test suite
- For search tasks: check if the retrieved page contains the expected answer
- For tool use: replay the tool calls in a sandbox and verify the outcome
Self-selection and majority voting are not reliable verifiers. List-wise ranking is marginally better but still far from oracle.
3. Design Context-Aware Pruning
If your agent runs more than 5 turns, consider context pruning between turns: summarize completed sub-tasks, drop failed intermediate reasoning, and surface only the most relevant prior context before the next turn. Several production frameworks (AutoGen Studio, LangChain's ConversationSummaryBufferMemory) offer this natively.
The paper's finding suggests that agents suffering from context ceiling are not failing because of insufficient reasoning capability — they are failing because the accumulated context is actively interfering. Pruning addresses this directly.
4. Track Domain Coverage in Evaluation
When reporting agent performance, be explicit about the task domain. A benchmark score from a coding-only dataset does not generalize to a general-purpose agent. Build evaluation sets that span the task types your agent will handle in production.
What Sequential Scaling Gets Right
<ul>
<li>Works well for well-scoped, single-domain tasks where feedback is clear</li>
<li>Allows agents to self-correct — useful for code debugging loops</li>
<li>Low implementation cost: just extend the loop limit</li>
</ul>
Where It Breaks Down
<ul>
<li>Context pollution from failed turns actively degrades later performance</li>
<li>No universal optimal turn count — requires per-task benchmarking</li>
<li>Cross-domain performance drops ~30% vs. domain-specific evaluation</li>
<li>Parallel sampling is only as good as your verifier</li>
</ul>
The Benchmark Itself
General AgentBench (https://github.com/cxcscmu/General-AgentBench) is publicly available. It provides:
- A unified evaluation framework across 4 task domains
- Pre-built wrappers for 10 LLM backends
- Scripts for running sequential and parallel scaling experiments
- Instructions for adding new benchmark tasks
Running the full evaluation requires API access to multiple LLMs and significant compute. The repo includes per-benchmark instructions in benchmarks/instructions/ for teams wanting to run individual domain evaluations.
How This Fits Into the Broader Test-Time Scaling Debate
The General AgentBench paper sits within a broader conversation about inference-time compute in 2026. OpenAI's o3 and o4 models, Anthropic's extended thinking, and Google's Gemini 3.0 Flash Thinking all demonstrate that additional compute at inference time can improve reasoning performance — for single-step reasoning tasks.
The key distinction the CMU paper draws is between reasoning models (where more thinking budget helps) and agentic systems (where more interaction turns often hurts). The failure modes are different:
- Reasoning models benefit from extended thinking because the thinking space is bounded and internally consistent
- Agentic systems accumulate external state (tool outputs, failed attempts, environmental responses) that can become contradictory or overwhelming
A paper from a separate group (arXiv:2506.12928, June 2026) confirms this distinction: test-time scaling for agentic coding specifically benefits from task decomposition and targeted retry rather than naive turn extension.
The consensus emerging from 2026 research: test-time compute scaling works, but the mechanism matters. Thinking harder on a single step works differently from trying again more times.
FAQ
Q: Does this mean I should never give agents more than 7 turns?
Not as a hard rule. The 3–7 range is a summary of the benchmark results, which used general-purpose LLMs on mixed-domain tasks. Specialized agents on narrowly scoped tasks (e.g., a coding agent working on unit test generation) can benefit from more turns if the context is well-structured. The point is to benchmark your specific use case rather than assuming more turns always helps.
Q: Which models held up best in the general evaluation?
The paper reports Claude showing the strongest robustness — the smallest performance gap between domain-specific benchmarks and the general evaluation. Most other models showed ~30% drops. The paper does not name every model, as evaluations were tied to API-accessible versions current at submission (Feb 2026).
Q: Is there a better alternative to list-wise ranking for the verification gap?
Yes: external execution. For coding tasks, running the generated code against a real test suite is dramatically more reliable than model-based selection. For search and tool-use tasks, running the proposed action in a sandboxed environment before committing is the equivalent. The paper's list-wise ranking is the best option when external verification is not available — not instead of it.
Q: How does this relate to speculative decoding and inference optimization?
Speculative decoding (e.g., Snowflake's Arctic Inference, which shows 1.6–1.8x speedup) addresses token generation latency within a single step. It does not affect multi-turn agent behavior. The context ceiling is a semantic problem (accumulated context quality), not a latency problem. These are orthogonal optimizations.
Q: Where can I read the full paper?
The paper is at arXiv:2602.18998. The benchmark code is at github.com/cxcscmu/General-AgentBench.
Key Takeaways
The CMU General AgentBench study delivers three findings that should directly influence how you design and evaluate agentic AI systems in 2026:
Sequential scaling has a ceiling: agent performance peaks between 3 and 7 turns on general tasks; beyond that, accumulated context pollution degrades results. Set explicit turn budgets.
Parallel scaling needs a verifier: sampling more trajectories raises the theoretical upper bound, but model self-selection cannot close the verification gap. Build external verifiers for code, tool use, and search tasks.
Domain-specific scores don't generalize: a strong SWE-bench or GAIA score does not predict cross-domain agent robustness. Evaluate on task distributions that match your production use case.
Bottom Line
The "just let it retry" approach to agentic AI has measurable limits. Context ceiling and verification gap are not edge cases — they are systematic properties of current LLM architectures under multi-turn workloads. Developers building production agent systems should treat turn budgets, context pruning, and external verification as first-class design decisions, not afterthoughts.
