The Planner Made the Same 3 Mistakes Every Time. A Bigger Model Didn't Fix It.

This is article 3 in a series about building PlannerCritic, an open-source engine where one LLM writes a plan and a second LLM reviews it. Article 1 covers the 157-goal field test. Article 2 is about the critic severity bug. This one is about the most uncomfortable thing the field test revealed: the planner has a structural problem that no model upgrade fixes.

132 concrete blockers across 63 strict goals. Three defect families. I tried gpt-4o. Same pattern. The fix is deterministic validation, not more parameters.

The Pattern

By the 10th strict goal, I noticed it. By the 50th, I could predict the failure before the critic printed it. By the 100th, I stopped being surprised and started being annoyed.

The planner kept making the same three mistakes. Not occasionally. Not randomly. Every single strict goal that failed did so because of one of three defect families.

Unverified dependencies (57 blockers): The plan declares a precondition that no earlier task establishes. The planner knows what should be true. It doesn't arrange the steps to make it true.

Real example — from ai-03-model-serving-migration:

[BLOCKER] unverified_dependencies — task=cutover_traffic_100
  "Cutover traffic from SageMaker to vLLM (100%) verification of
  latency SLO is dependent on prior traffic cutover stages being
  established but lacks clear confirmation of stability before
  proceeding."

The plan says "cutover 100% of traffic" but no earlier task verifies that the 10% and 50% stages were stable.

Unsafe sequencing (46 blockers): Tasks are ordered before their hard prerequisites. The cutover runs before the verification. The backup completes after the migration.

Real example — from ai-02-embedding-index-migration:

[BLOCKER] unsafe_sequencing — task=backfill_vectors
  "Backfill operation cannot proceed until the index is verified
  for quality; it is ordered incorrectly in the sequence."

The plan puts the backfill before the quality check that should gate it.

Weak rollback (18 blockers): High-blast-radius steps lack rollback. The planner includes rollback on routine steps but omits it on cutover, teardown, and failback.

Real example — from db-10-multi-tenant-split:

[BLOCKER] weak_rollback — task=dual_write_setup
  "The dual-write setup task's rollback only switches to
  single-write mode without addressing potential inconsistencies
  during the transition."

The rollback exists but it is not credible. Switching back to single-write does not undo the data inconsistencies that dual-write may have introduced.

I Tried a Bigger Model

The natural instinct: use gpt-4o. I kept waiting for it to bail me out.

I tested it both ways — gpt-4o planner with mini critic, and gpt-4o for both roles.

Same defect pattern. Better prose. Same structural mistakes.

Unverified dependencies. Unsafe sequencing. Weak rollback.

That was the moment I stopped blaming the model size. The planner wasn't dumb. That was the annoying part. It was plausible. It knew the right steps. It just couldn't close the dependency graph or enforce the ordering.

I did not have a smaller-model problem. I had a planning-structure problem.

Why the Loop Can't Fix It

The revision loop is designed to converge. The critic reports blockers. The planner revises.

But the planner tends to fix one blocker and introduce another. It reshuffles task order without closing the dependency gap. It adds rollback to the wrong task.

After 2 revisions — the median across 33 strict goals — the planner stops making meaningful changes. The convergence detector fires. The engine escalates.

The loop works as designed. The planner is the bottleneck. I kept expecting the revision loop to converge. It didn't, because the planner can't fix a structural problem by rewriting the prose.

Worth being clear about what failed here. The critic reliably found the same blockers across revisions — the failure was the planner's inability to structurally repair, not the critic's judgment. That's a different defect from the one in Article 2, where the critic's severity calibration was the problem.

The Fix Is Not More Parameters

The highest-leverage fix is a precondition closer: a deterministic linter that runs after the planner produces a draft and verifies that every precondition is actually established by an earlier task. If a task says "requires replica_verified," there must be a prior task that produces it.

This single pass would eliminate 64 of 132 blockers (48%) without asking the LLM to get smarter.

The remaining blockers — unsafe sequencing and weak rollback — need either better prompt engineering, additional deterministic validation, or genuinely better reasoning about ordering and risk.

What the Research Says

This is not just my observation. The academic literature converges on the same finding.

The "Why Reasoning Fails to Plan" paper (arXiv 2601.22311) shows that LLM agents select actions based on local evaluation without considering future consequences. In knowledge graph traversals, single-step greedy policies select myopic traps more than 55% of the time. The authors prove step-wise reasoning is provably insufficient for long-horizon planning.

The PlanGenLLMs survey (arXiv 2502.11221) evaluates LLM planning across four criteria — completeness, executability, optimality, representation. LLMs consistently fail at ensuring plans are executable: preconditions not met, steps out of order.

The field is converging on hybrid approaches — LLM plus deterministic validation plus classical planning techniques. Not LLM alone.

What You Should Take Away

If you are building agents that plan over multiple steps:

Test your planner on a real corpus. The 157-goal run revealed a pattern that 3 demo goals would never show.
Measure defect types, not just pass/fail. If all your failures are in one family, you have a specific gap.
Do not assume a bigger model will fix structural problems. The planning gap is a reasoning limitation, not a language ability limitation.
Add deterministic validation before trusting the LLM to self-correct. The revision loop is useful, but it is not a substitute for structural checks.

Article 3 of 5 in the PlannerCritic series.

Series: Article 1: "I Ran 157 Agent Plans Against a Real LLM" · Article 2: "I Told My LLM Critic to Be Adversarial" · Article 4: "The Field Test Found 10 Issues" · Article 5: "I Tried to Prompt-Inject My Own Engine"

Links: