From Goroutines to Agents: Lessons from 1M Concurrent Threads and the New Wave of AI Engineering

Originally published on tamiz.pro.

In 2021, a team running a Go-based infrastructure service hit 1 million concurrent goroutines under load. The lessons they pulled from that scale—structured concurrency, cancellation propagation, resource budgeting, observable failure—landed differently this time around. Today, the same patterns are surfacing as engineers build production AI agent systems, except instead of goroutines racing on an event loop, we're managing LLM calls, tool executions, and streaming responses across distributed services.

The parallel isn't coincidental. Both domains share a core tension: unbounded fan-out looks elegant in code and catastrophic in production. Understanding how the Go community solved this at massive scale gives AI engineers a head start on the problems that are now hitting agent platforms.

1. The Fan-Out Problem: When Parallelism Becomes Chaos

Goroutine Leak at Scale

The 1M-goroutine incident started innocently. A request handler spawned a worker goroutine per downstream call:

func HandleRequest(ctx context.Context, req Request) ([]Result, error) {
    var wg sync.WaitGroup
    results := make([]Result, len(req.Subtasks))

    for i, task := range req.Subtasks {
        wg.Add(1)
        go func(idx int, t Task) {
            defer wg.Done()
            results[idx] = process(ctx, t)
        }(i, task)
    }

    wg.Wait()
    return results, nil
}

Under normal load this was fine. But when the system saw a burst of requests—each with 50–200 subtasks—the goroutine count spiked. Without cancellation on ctx, every in-flight goroutine survived until its upstream request timed out or the process was killed. The Go runtime didn't crash (it's designed for this), but the scheduler overhead became significant, and memory from pending operations accumulated.

The fix wasn't removing goroutines—it was adding boundaries:

func HandleRequest(ctx context.Context, req Request) ([]Result, error) {
    // Bound concurrency, not just spawn freely
    sem := make(chan struct{}, 500) // max 500 concurrent workers

    var wg sync.WaitGroup
    results := make([]Result, len(req.Subtasks))

    for i, task := range req.Subtasks {
        wg.Add(1)
        go func(idx int, t Task) {
            defer wg.Done()
            select {
            case sem <- struct{}{}:
                defer func() { <-sem }()
                results[idx] = process(ctx, t)
            case <-ctx.Done():
                return
            }
        }(i, task)
    }

    wg.Wait()
    return results, nil
}

The Agent Equivalent

AI agent frameworks exhibit the exact same pattern today. Consider a typical "plan-and-execute" agent:

# Common anti-pattern in agent frameworks
for subtask in plan.subtasks:
    result = await execute_agent(subtask)  # Unbounded fan-out!
    results.append(result)

A single user request can fan out to 20–50 parallel LLM calls, each with its own context window, API latency, and error surface. Without concurrency limits, you're hitting rate limits, blowing your token budget, and degrading response quality for all concurrent users. The goroutine leak becomes a token leak and a latency cascade.

The parallel solution is identical: semaphore-based bounded concurrency, context propagation, and structured cleanup:

from asyncio import Semaphore

async def run_plan(plan: Plan, max_concurrent: int = 10) -> list[Result]:
    sem = Semaphore(max_concurrent)

    async def bounded_execute(subtask: Subtask) -> Result:
        async with sem:
            return await execute_agent(subtask)

    tasks = [bounded_execute(t) for t in plan.subtasks]
    results, _ = await asyncio.wait(tasks, timeout=plan.timeout)
    return [r.result for r in results]

The lesson scales: whether it's goroutines, HTTP connections, or LLM invocations, unconstrained parallelism is a design smell. The Go community learned this the hard way at scale. AI engineers are learning it now—often before the hard part.

2. Cancellation: The Silent Killer of Correctness

How Goroutines Handle Abandonment

Go's context package is a masterclass in cooperative cancellation. When a parent context is cancelled, all descendants must observe that signal and stop work:

func deepProcess(ctx context.Context, work Work) error {
    // Every internal call must respect ctx
    step1Data, err := stepOne(ctx, work.Input)
    if err != nil {
        return fmt.Errorf("step one failed: %w", err)
    }

    childCtx, cancel := context.WithCancel(ctx)
    defer cancel() // Always clean up

    go func() {
        select {
        case <-childCtx.Done():
            return
        default:
            step2Data := stepTwo(childCtx, step1Data)
            writeResult(step2Data)
        }
    }()

    return nil
}

The critical invariant: every goroutine in the call tree inherits a context that can be cancelled. If any link in the chain drops the context and spawns an uncontrolled goroutine, you have a leak. The Go runtime doesn't enforce this—you do.

What This Means for Agents

n
AI agent pipelines face the same cancellation problem, but it's harder to see. When a user cancels a long-running agent, you need to propagate that signal through:

1. In-flight LLM API calls — Must be aborted (or at least ignored when responses arrive)
2. Tool execution goroutines — Must respect cancellation
3. Callback/update streams — Must stop pushing results to dead handlers
4. State mutations — Should be rolled back or abandoned cleanly

from dataclasses import dataclass
from typing import Optional
import asyncio

@dataclass
class AgentContext:
    cancelled: asyncio.Event
    budget: TokenBudget
    steps: list[StepResult]

async def agent_step(
    ctx: AgentContext,
    plan: AgentPlan,
    llm_client: LLMClient,
) -> AgentResult:
    """Each step must observe cancellation and budget."""
    for subtask in plan.sequence:
        # Check cancellation before each step
        if ctx.cancelled.is_set():
            return AgentResult(status="cancelled", steps=ctx.steps)

        # Check token budget
        cost = await llm_client.estimate_cost(subtask.prompt)
        if not ctx.budget.can_spend(cost):
            return AgentResult(status="budget_exceeded", steps=ctx.steps)

        result = await llm_client.generate(
            prompt=subtask.prompt,
            cancel_event=ctx.cancelled,  # Pass cancellation down
        )
        ctx.steps.append(result)
        ctx.budget.spend(result.usage.tokens)

    return AgentResult(status="completed", steps=ctx.steps)

The insight from the Go community: cancellation isn't an error handling concern—it's a correctness concern. An agent that continues producing results after the user has moved on isn't just wasteful; it's actively harmful if those results feed back into state that another request reads.

3. Structured Concurrency: From Channels to Tool Calling

The Go Philosophy

Go's guiding principle for concurrency is well-known: "Do not communicate by sharing memory; share memory by communicating." Channels enforce ordering, prevent races, and make the control flow explicit. A goroutine that receives on a channel will block until data arrives or the channel closes—it doesn't spin, poll, or guess.

// Clean pipeline: producer → processor → consumer
func pipeline(ctx context.Context) error {
    tasks := make(chan Task)
    results := make(chan Result)

    go producer(ctx, tasks)
    go processor(ctx, tasks, results)
    go consumer(ctx, results)

    <-ctx.Done()
    return ctx.Err()
}

Each stage communicates through typed channels. There are no shared mutable variables between stages. The lifecycle of each goroutine is tied to a channel close or a context cancellation.

Agent Frameworks Are Evolving Toward This

The new wave of production agent frameworks—CrewAI, AutoGen, LangGraph, custom builds—are converging on the same structured patterns. Instead of arbitrary function calls between agents, they use tool graphs where each node's inputs and outputs are explicitly declared:

from langgraph.graph import StateGraph, END

# Define the graph structure explicitly
workflow = StateGraph(AgentState)

# Each node is a named function with typed inputs/outputs
workflow.add_node("researcher", researcher_agent)
workflow.add_node("analyst", analyst_agent) 
workflow.add_node("writer", writer_agent)
workflow.add_node("reviewer", reviewer_agent)

# Edges define the control flow—no mystery
workflow.add_conditional_edges(
    "researcher",
    should_analyze,  # conditional routing
    {"analyze": "analyst", "done": END}
)
workflow.add_edge("analyst", "writer")
workflow.add_conditional_edges(
    "writer",
    needs_review,
    {"review": "reviewer", "done": END}
)

app = workflow.compile()

This is the agent equivalent of Go channels: data flows through explicit conduits, and control is visible in the graph structure, not hidden in side effects. The difference is that Go channels are synchronous by default (or explicitly made async), while agent graphs often mix synchronous and asynchronous I/O in ways that require careful coordination.

4. Resource Budgeting: Why "Just More Concurrency" Breaks Everything

The OOM Scenario

At 1M goroutines, the team didn't hit an OOM error from goroutine memory alone (each goroutine starts at 2KB). They hit it from stack growth under load combined with buffered channels that accumulated unprocessed messages. The runtime could schedule them, but the heap couldn't keep up with allocation velocity.

The real fix wasn't reducing goroutine count—it was reducing the work each goroutine did and ensuring every goroutine had a bounded scope:

// Bounded work per goroutine: one task, done, exit
func worker(ctx context.Context, tasks <-chan Task, results chan<- Result) {
    for {
        select {
        case task, ok := <-tasks:
            if !ok {
                return // Channel closed, exit gracefully
            }
            result := doOneTask(ctx, task)
            select {
            case results <- result:
            case <-ctx.Done():
                return
            }
        case <-ctx.Done():
            return
        }
    }
}

Each worker has a clear lifecycle: enter, process one item, exit. No accumulation. No state carrying over between calls.

Agent Resource Budgets

AI agents face analogous resource constraints—token budgets, API rate limits, and reasoning depth budgets (how many steps before you decide you're stuck):

class AgentBudget:
    def __init__(self, max_tokens: int, max_steps: int, max_time_seconds: float):
        self.max_tokens = max_tokens
        self.max_steps = max_steps
        self.max_time = max_time_seconds
        self.tokens_spent = 0
        self.steps_taken = 0
        self.start_time = time.monotonic()

    def check(self) -> BudgetStatus:
        elapsed = time.monotonic() - self.start_time
        if elapsed > self.max_time:
            return BudgetStatus.TIMEOUT
        if self.steps_taken >= self.max_steps:
            return BudgetStatus.MAX_STEPS
        if self.tokens_spent >= self.max_tokens:
            return BudgetStatus.TOKEN_EXHAUSTED
        return BudgetStatus.OK

    def spend(self, tokens: int) -> bool:
        self.tokens_spent += tokens
        self.steps_taken += 1
        return self.check() == BudgetStatus.OK

Every agent step checks the budget before proceeding. This isn't paranoia—it's the difference between an agent that produces a useful answer in 8 steps and one that loops 47 times burning $12 in API costs.

The parallel is direct: goroutine budgeting prevented scheduler thrash; token/step budgeting prevents reasoning loops. Both are about constraining per-unit resource consumption, not just total concurrency.

5. Observability: You Can't Fix What You Can't See

Go's/pprof Contribution to Production Engineering

One of Go's greatest gifts to production engineering is profiling. The standard library includes net/http/pprof out of the box, giving you goroutine dumps, heap profiles, blocking profiles, and mutex contention data with a single import:

import _ "net/http/pprof"

// That's it. Now /debug/pprof/goroutine gives you
// a full snapshot of every goroutine, its stack trace,
// and what it's waiting on.
http.ListenAndServe(":6060", nil)

At 1M goroutines, the team used blocking profiles to discover that 30% of goroutines were blocked on channel sends to a single congested output channel. The fix was fan-out: replacing one shared channel with N per-worker channels that a single aggregator goroutine collected.

Distributed Tracing for Agents

AI agent systems need equivalent visibility, but the profiling surface is different:

# Example: OpenTelemetry tracing for an agent pipeline
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("agent.pipeline")

async def run_agent(user_request: str) -> AgentOutput:
    with tracer.start_as_current_span("agent.execution") as span:
        span.set_attribute("user.id", user_request.metadata.user_id)
        span.set_attribute("agent.type", "research-agent")

        # Trace each step
        plan = await generate_plan(user_request)
        span.set_attribute("plan.steps", len(plan.subtasks))

        results = []
        for i, subtask in enumerate(plan.subtasks):
            with tracer.start_as_current_span(f"agent.step.{i}") as step_span:
                step_span.set_attribute("step.index", i)
                step_span.set_attribute("step.type", subtask.type)

                result = await execute_with_budget(subtask, budget)

                if result.error:
                    step_span.set_status(Status(StatusCode.ERROR))
                    step_span.record_exception(result.error)

                results.append(result)

        return AgentOutput(results=results, total_tokens=sum(r.tokens for r in results))

With this trace data, you can answer: which step is consuming the most tokens? Which is blocked waiting for LLM responses? How many agent invocations are in-flight concurrently? These are the agent-equivalent of "how many goroutines are blocked on which channel."

6. Failure Domains and Graceful Degradation

The Go Lesson: Fail Closed, Not Open

When the 1M-goroutine system encountered failures, the engineers learned to fail closed: if one worker failed, the whole pipeline shouldn't silently continue producing garbage results. Instead, the pipeline should either complete with partial results or fail fast with a clear error.

func processAll(ctx context.Context, tasks []Task) ([]Result, error) {
    type taskResult struct {
        index int
        result Result
        err    error
    }

    resultsCh := make(chan taskResult, len(tasks))

    var wg sync.WaitGroup
    for i, task := range tasks {
        wg.Add(1)
        go func(idx int, t Task) {
            defer wg.Done()
            result, err := doWork(ctx, t)
            resultsCh <- taskResult{idx, result, err}
        }(i, task)
    }

    go func() {
        wg.Wait()
        close(resultsCh)
    }()

    outputs := make([]Result, len(tasks))
    var firstErr error

    for r := range resultsCh {
        if r.err != nil && firstErr == nil {
            firstErr = fmt.Errorf("task %d failed: %w", r.index, r.err)
        }
        outputs[r.index] = r.result
    }

    if firstErr != nil {
        return nil, firstErr // Fail closed: no partial results on error
    }
    return outputs, nil
}

Agent Degradation Strategies

AI agents need equally principled failure handling. The stakes are different—agents don't crash your program, but they can generate plausible-sounding but incorrect outputs (hallucinations) or waste resources on unproductive loops. Common degradation patterns:

class AgentFallback:
    """Strategies for when an agent pipeline fails."""

    @classmethod
    async def handle_step_failure(cls, step: Step, error: Exception) -> StepResult:
        match error:
            case LLMRateLimitError():
                # Exponential backoff, don't fail the whole pipeline
                return await cls.retry_with_backoff(step, error)

            case LLMTimeoutError():
                # Use a cheaper/faster model as fallback
                return await cls.fallback_to_quick_model(step)

            case BudgetExceededError():
                # Return partial results collected so far
                return StepResult(
                    status="partial",
                    output=cls.summarize_partial(step.context),
                    warning="token budget exhausted mid-pipeline"
                )

            case HallucinationDetectionError():
                # Re-prompt with stronger constraints
                return await cls.retry_with_constraints(step, strict=True)

            case MaxRetriesExceeded():
                # Escalate: return an error the user can act on
                return StepResult(
                    status="failed",
                    error=AgentPipelineError(f"step {step.id} failed after retries"),
                    recovery_hint="try simplifying the prompt or reducing subtask complexity"
                )

The key principle, learned from systems that ran at massive concurrency: failure modes should be observable, bounded, and recoverable. A goroutine that panics without recovery brings down the process. An agent that hallucinates without guardrails brings down trust in the system.

7. The Mental Model Shift: From Resources to Outcomes

What 1M Goroutines Taught Us

Running a million concurrent units of work doesn't make you a concurrency expert. It makes you learn the hard way that:

• Concurrency is a cost, not a free optimization. Each concurrent unit consumes scheduler time, memory, and coordination overhead.
• Boundedness is safety. Unbounded concurrency is a design flaw that manifests under load.
• Observability is non-optional at scale. If you can't profile it, you can't improve it.
• Cancellation is a feature, not an afterthought. Systems that don't support clean shutdown are systems that leak.

What This Means for AI Agent Engineering

The same principles apply to agents, which are themselves concurrent units of work that happen to call LLMs:

The shift isn't that the problems changed—it's that the abstraction surface grew. A goroutine is a thread of execution. An agent is a thread of execution that can reason, call tools, and invoke other agents. The concurrency principles are the same; the failure modes are just richer.

Frequently Asked Questions

Q: Are AI agents really comparable to goroutines, or is the analogy overstated?
A: The analogy holds at the systems level—both are lightweight concurrent units whose lifecycle must be managed. The difference is that agents have semantic state and can make decisions, while goroutines don't. But the resource management problems (cancellation, bounding, observability, failure handling) are structurally identical. Treating agents as "just another concurrent thing" rather than something qualitatively different is often the right engineering instinct.

Q: Do I need to implement all of these patterns from scratch for my agent system?
A: No. Modern agent frameworks like LangGraph, CrewAI, and AutoGen already encode many of these patterns. The value is in understanding why they exist—so when a framework doesn't cover your edge case (and it won't), you know which principle to reach for. For example, if your framework doesn't support step-level token budgeting, you add it using the AgentBudget pattern above.

Q: What's the single most important lesson from the 1M-goroutine experience for agent builders?
A: Measure before you optimize, and bound before you scale. The teams that survived 1M goroutines didn't start with 1M goroutines—they started with 1,000, measured everything, and added concurrency only where the data showed it helped. The same applies to agents: build with bounded concurrency and rich observability from day one. Don't assume you'll remember to add budgets and traces later.

For more on building production-grade AI systems, explore the engineering insights at tamiz.pro, which covers agent architectures, concurrency patterns, and real-world lessons from shipping AI systems at scale.