Planning Over Execution: Lessons from 157 Agent Runs and the Rise of Orca-Style Agent Fleets

Originally published on tamiz.pro.

The field of AI agents has moved rapidly from single-model executors to complex multi-agent orchestration. But after running 157 agent deployments across diverse task domains, one pattern emerged with striking consistency: planning quality predicts success far better than execution speed or model size. This isn't just theoretical—it's a practical lesson that's reshaping how engineers architect agent fleets, giving rise to what we're now calling Orca-style agents: hierarchical, planning-first systems that separate the expensive business of thinking from the cheaper business of doing.

The Experiment: 157 Agent Runs

Over six months, our team deployed and monitored 157 distinct agent runs across four primary use cases: code generation pipelines, automated testing workflows, infrastructure-as-code provisioning, and data transformation tasks. Each run varied along three dimensions:

• Architecture: Single-agent vs. flat multi-agent vs. hierarchical (Orca-style)
• Planning depth: No planning, brief intent statement, or full recursive planning loop
• Execution model: Direct LLM call per action vs. tool-augmented execution with validation

The results were unambiguous. Systems that invested 3-5x more tokens in planning achieved 4.2x higher task completion rates and 3.8x fewer rollback cycles compared to agents optimized purely for fast execution. The correlation between planning sophistication and success held across every domain.

Why Planning Beats Raw Execution

The intuition behind this finding rests on an economic principle of LLM usage: planning is cheap relative to costly mistakes. A well-structured plan reduces the probability of executing the wrong sequence of tools, making incorrect API calls, or generating code that fails integration testing.

Consider the token economics:

Agents that plan thoroughly front-load their costs. Those that rush to execute often pay exponentially more in corrections, retries, and human intervention.

The Orca Architecture Pattern

The name "Orca" comes from the hierarchical social structure of killer whales: a single matriarch orchestrates, while specialized pod members execute discrete tasks. In agent terms, this translates to:

Core Components

1. Strategic Planner (the matriarch): Holds global context, decomposes goals, assigns subtasks, and validates outcomes. Runs on a stronger model with longer context windows.

2. Specialist Executors (the pod): Each handles a narrow domain—code generation, test writing, documentation, validation. Run on smaller, cheaper models optimized for throughput.

3. Shared Memory Layer: A structured knowledge graph or vector store that maintains state across the fleet, preventing redundant work and enabling cross-agent learning.

4. Orchestration Loop: A lightweight controller that routes tasks, aggregates results, and triggers replanning when validation fails.

Why This Separation Matters

The critical insight is that not all thinking is equal. Strategic decisions—understanding requirements, identifying edge cases, sequencing dependencies—benefit from deep context and reasoning. Tactical decisions—formatting output, calling a specific API, generating a template—are better handled by focused, optimized models.

Separating these concerns allows you to:

• Run planners on premium models without paying premium prices for every action
• Scale executor capacity independently of planning capacity
• Implement targeted retry logic without restarting entire workflows
• Observe and debug planning failures separately from execution failures

Implementation Patterns

From the 157 runs, several implementation patterns emerged as particularly effective:

Pattern 1: Recursive Decomposition with Validation Gates

The planner decomposes a goal into subtasks, each with explicit success criteria. Executors complete subtasks and return structured evidence of completion. The planner validates before proceeding to the next level.

class OrcaPlanner:
    async def decompose(self, goal: str, context: Dict) -> List[Subtask]:
        """Recursive planning with validation gates."""
        plan = await self.model.plan(goal, context=context)

        validated_subtasks = []
        for subtask in plan.subtasks:
            if subtask.requires_decisions():
                # Recursive planning for complex subtasks
                sub_plan = await self.decompose(subtask.description, context)
                validated_subtasks.extend(sub_plan)
            else:
                validated_subtasks.append(subtask)

        return validated_subtasks

Pattern 2: Specialist Routing with Skill Cards

Each executor carries a "skill card"—a concise description of its capabilities, constraints, and preferred input/output formats. The planner matches subtasks to specialists based on these cards rather than attempting blind routing.

Pattern 3: Stateful Context Propagation

Instead of passing raw conversation history, agents exchange structured context objects: constraints discovered, assumptions made, partial results, and confidence scores. This enables better downstream planning and reduces context window waste.

Common Pitfalls from the 157 Runs

Not every design decision landed well. Here are the patterns that correlated with failure:

Pitfall 1: Over-Planning

Some teams spent so much time planning that the plan became stale before execution began. The sweet spot was ~20% of total token budget for planning, with dynamic replanning triggered only by validation failures—not on a timer.

Pitfall 2: Specialist Fragmentation

Creating too many specialists (15+) introduced routing overhead and context fragmentation. The optimal range was 4–8 specialists, each covering a distinct capability domain.

Pitfall 3: Silent Replanning

When execution failed, some systems silently retried with minor variations. Successful systems explicitly logged failures, triggered replanning at the appropriate abstraction layer, and maintained an audit trail.

Pitfall 4: Context Window Hoarding

Planning agents that retained full conversation history from execution agents burned through context windows unnecessarily. Successful implementations used summarized state objects instead.

When to Use Orca-Style Fleets

These systems aren't a universal upgrade. Based on the data:

The break-even point appears around 3–5 sequential steps with dependency checking. Below that, the planning overhead outweighs the benefits.

The Economics of Planning-First Design

For engineers evaluating whether to adopt Orca-style architectures, the cost model matters:

Traditional approach: One large model handles everything. Cheap for simple tasks, expensive for complex ones due to rework.

Orca approach: Premium model for planning, lighter models for execution. Higher upfront cost, dramatically lower correction costs.

In our measurements, the Orca architecture showed a 37% reduction in total token cost for tasks exceeding 10 steps, despite using more expensive models for planning. The savings came from reduced retry loops and fewer human escalations.

Looking Ahead

The 157-agent study reveals a fundamental shift in how we should think about agentic systems. The question is no longer "how fast can the agent execute?" but "how well can the agent plan?" This reframing is driving a new generation of tools focused on planning quality: structured decomposers, validation-aware schedulers, and context-efficient state management.

As the field matures, we'll likely see planning becomes a first-class concern in agent frameworks, with libraries and patterns emerging specifically for the strategic layer. The Orca architecture isn't just a pattern—it's an acknowledgment that thinking carefully is the most important part of the job.

Frequently Asked Questions

Q: Can I implement Orca-style planning without a full multi-agent framework?
A: Yes. Start with a single planner function that decomposes goals and validates each step before execution. You don't need separate agents initially—a well-structured planning loop within a single process captures 80% of the benefit.

Q: How do I choose which model to use for planning vs. execution?
A: Planning benefits from strong reasoning and long context (e.g., GPT-4-class models). Execution can use smaller, faster models optimized for specific tasks. The key is matching model capability to cognitive demand, not cost alone.

Q: What's the minimum viable Orca architecture?
A: Three components: (1) a planner that decomposes goals into sequenced subtasks with success criteria, (2) an executor that runs subtasks and returns structured results, and (3) a validator that checks completion evidence before allowing progression. This fits in under 200 lines of code.