Originally published on Loop & Retry — field notes on building LLM agents that survive production.
Here's the most useful thing I can tell you about agent architecture: most of the time, don't build one. The task in front of you probably has a known set of steps, and a thing with a known set of steps is a pipeline, not an agent — building it as an agent buys you nondeterminism, latency, and a token bill you didn't need, in exchange for flexibility you're not going to use.
This post is the decision I make before writing any orchestration code: does this task actually need an agent, or is it a fixed workflow wearing a costume? I'll define the line precisely, show the arithmetic on what autonomy costs, build one task both ways so the difference is concrete, and end with the checklist I actually run down.
First, a definition that does real work
The word "agent" has been stretched to mean "anything with an LLM in it," which makes the design question impossible to reason about. So here's the distinction I use, and it's the one that matters for cost and reliability:
- A workflow is an LLM (or several) orchestrated through control flow you wrote. The code decides what happens next. The model fills in the steps; the sequence is fixed.
- An agent is an LLM that decides its own control flow. It's a model in a loop with tools, and the model — not your code — chooses which tool to call next and when to stop.
That single property — who owns the control flow — is the whole decision. When your code owns it, you can read the path, test the path, and bound the cost of the path. When the model owns it, you've traded all three away for the ability to handle situations you couldn't enumerate in advance. Sometimes that trade is exactly right. Usually the situations were enumerable and you just hadn't written them down yet.
The agent tax
Autonomy isn't free, and the cost isn't abstract. Four things get worse the moment the model owns the loop.
Token cost goes quadratic. This is the one people underestimate. In an agent loop, each step re-sends the entire conversation so far — system prompt, tool schemas, and every prior turn and tool result. If each step adds roughly a constant amount of context, then step k sends about k units, and N steps send 1 + 2 + … + N ≈ N²/2 units total. A 10-step agent doesn't cost 10× a single call; the input side costs closer to 50×. A single structured call sends the context once. This is exactly the cost model I explored in how agents with long trajectories compound their cost.
Put numbers on it. Say your base context (system prompt + tool schemas + input) is 4,000 tokens, and each step appends ~800 tokens of assistant reasoning and tool output. A one-shot call reads 4,000 input tokens. A 10-step agent reads 10×4000 + 800×(0+1+…+9) = 40,000 + 36,000 = 76,000 input tokens for the same job — 19× the reads, before you count a single output token. Prompt caching claws some of this back for the stable prefix, but the part that grows every step — the transcript — is exactly the part caching helps least.
Latency is serial. Every tool call in an agent loop is a round trip: model → tool → model → tool. Ten steps is ten sequential model calls plus ten tool executions, and you can't parallelize a sequence where step k depends on the result of step k−1. A workflow with known structure can fan out independent calls concurrently; an agent that discovers its plan one step at a time cannot.
The failure surface is the whole trajectory. A single call fails in one place. A ten-step agent can go wrong at any step, and — worse — a wrong-but-plausible intermediate result poisons every step after it. When it fails you're not debugging a function, you're debugging a path that was different last time. These failures are often silent — the agent completes, returns exit zero, and the result is wrong. (This is exactly why grading an agent means grading a trajectory, not an output — I wrote a whole post on why that's hard.)
You can't unit-test control flow you don't own. assert route(ticket) == "billing" is a test. There is no clean assertion for "the agent will, across runs, choose a reasonable sequence of tool calls," because the sequence is a distribution, not a value. You can evaluate it statistically over a suite, but you've left the world of cheap deterministic tests — and you left it voluntarily.
None of this is an argument against agents. It's an argument for making sure you're buying something with it.
The task that doesn't need an agent (but often gets one)
Support-ticket triage: read a ticket, classify it, pull the right canned next-step. I've seen this built as an agent — model, tool belt, while loop — because "agent" is the default shape now. Here's that version, using the Anthropic SDK (anthropic==0.40.0, model claude-sonnet-4-6):
import anthropic
client = anthropic.Anthropic()
TOOLS = [
{"name": "lookup_account", "description": "Get account tier for a user",
"input_schema": {"type": "object", "properties": {"user_id": {"type": "string"}},
"required": ["user_id"]}},
{"name": "get_playbook", "description": "Fetch the response playbook for a category",
"input_schema": {"type": "object", "properties": {"category": {"type": "string"}},
"required": ["category"]}},
]
def triage_agent(ticket, user_id):
messages = [{"role": "user",
"content": f"Triage this ticket for user {user_id}. "
f"Classify it, look up whatever you need, and return the playbook.\n\n{ticket}"}]
while True: # the model owns the loop — and the cost, and the failure modes
resp = client.messages.create(
model="claude-sonnet-4-6", max_tokens=1024, tools=TOOLS, messages=messages)
messages.append({"role": "assistant", "content": resp.content})
if resp.stop_reason != "tool_use":
return resp # model decided it's done — whenever that is
results = []
for block in resp.content:
if block.type == "tool_use":
out = run_tool(block.name, block.input) # your dispatch
results.append({"type": "tool_result", "tool_use_id": block.id, "content": out})
messages.append({"role": "user", "content": results})
Look at what you've signed up for. The number of loop iterations is a model decision, so your cost per ticket is a distribution — usually 2–3 calls, occasionally 6 when it second-guesses a classification, and there's no hard ceiling unless you add one. The stop condition is "the model stopped asking for tools," which is not the same as "the ticket is correctly triaged." And to test it you have to run it, because the path isn't in your code.
But look at the task itself: the steps are fixed. Classify → maybe look up the account → fetch the playbook. You know that sequence at design time. You wrote it in the docstring. So write it in code:
from pydantic import BaseModel
from typing import Literal
class Triage(BaseModel):
category: Literal["billing", "bug", "howto", "abuse"]
urgency: Literal["low", "normal", "high"]
needs_account_lookup: bool
def triage_pipeline(ticket, user_id):
# ONE structured call. Control flow is yours; the model just fills the slots.
resp = client.messages.create(
model="claude-sonnet-4-6", max_tokens=512,
tools=[{"name": "classify", "input_schema": Triage.model_json_schema()}],
tool_choice={"type": "tool", "name": "classify"}, # forced: exactly one call, no loop
messages=[{"role": "user", "content": ticket}],
)
t = Triage(**next(b.input for b in resp.content if b.type == "tool_use"))
account = lookup_account(user_id) if t.needs_account_lookup else None # your branch, not the model's
playbook = get_playbook(t.category) # deterministic
return {"triage": t, "account": account, "playbook": playbook}
Same capability. But the control flow is code you can read and test (assert triage_pipeline(billing_ticket, u).triage.category == "billing"), the account lookup happens on a branch you control, the playbook fetch is a dict lookup with zero model involvement, and the cost is exactly one bounded model call per ticket. Forcing tool_choice to a single tool turns the "agent" back into a function. You didn't lose anything, because the flexibility the agent offered — deciding the steps at runtime — was flexibility this task never needed.
The tell is general: if you can write the sequence of steps in the docstring, it belongs in the code, not in the model's head.
When you actually do need one
To be fair to agents, here's the flip side — the shape of a task that earns the tax.
You're building a coding assistant that, given "the integration test is flaky, fix it," has to: read the test, form a hypothesis, grep for the relevant source, read it, maybe run the test to confirm the failure, edit, re-run, and iterate until green. You cannot write that sequence in advance. How many files it reads, whether it needs to run the test twice or five times, which functions it greps for — all of it depends on what it finds along the way. The branching factor is enormous and the path is genuinely data-dependent.
That's the real signature of an agent-shaped task, and it's narrower than the hype implies:
- The steps aren't enumerable in advance. Not "long," but genuinely unknowable — the next action depends on the content of prior observations in a way you can't flatten into branches.
- The action space is open-ended. The set of useful next moves is large and context-dependent, not a fixed menu of three.
- Feedback is available mid-task. Tests, compilers, search results — the environment can tell the agent whether it's on track, so the loop has something to correct against. An agent with no mid-run signal is just an expensive way to guess.
- The value justifies the variance. You're willing to accept nondeterministic cost and latency because a correct autonomous solution is worth much more than a cheap deterministic wrong one.
If you can't check off most of that list, you have a workflow. And there's a whole middle ground worth naming: workflows with LLM steps — prompt chains, routing, parallel fan-out, evaluator-optimizer loops with a fixed structure. These get you most of the "AI-powered" capability with almost none of the agent tax, because your code still owns the control flow. Reach for the agent only when the control flow genuinely has to be discovered at runtime.
The checklist I actually run
Before I build anything as an agent, I answer these. Every "no" pushes me toward a workflow or a single call:
- Can I write the steps in advance? If yes → pipeline. Put the sequence in code.
- Is the action space a small fixed menu? If yes → a router (one classification call) plus deterministic branches.
- Is there real feedback mid-task for the loop to correct against? If no → an agent is just guessing in a loop; use a single well-prompted call.
- Can I bound the cost? If I can't state a hard step cap and a per-run token ceiling, I'm not ready to run it anywhere near production. (This is a retry budget by another name — the same discipline that stops a retry loop from running forever is what stops an agent loop from doing it.)
- Would a wrong intermediate step be caught? If a plausible-but-wrong step silently poisons the rest, the trajectory needs checkpoints — or it needs to not be an agent.
- Is the flexibility worth the tax? If the deterministic version does the job, the burden of proof is on the agent to justify its cost, not the other way around.
The default in this space is to reach for the most capable, most autonomous architecture available and scale down only when forced. Invert it. Start with a single call. Add structure — a chain, a router, fixed branches — only when the task demands it. Hand the control flow to the model only when you genuinely cannot write it yourself. That's not a limitation on what you can build; it's how you keep the thing debuggable, affordable, and testable while it's still small enough to get right. The best agent is often the one you didn't build.
