Originally published on Loop & Retry — field notes on building LLM agents that survive production.
Running the context window as a cache — admit, evict, summarize, reorder — fixes the append-only habit for runs of tens of steps. It does not fix it for runs of hundreds or thousands, and long-running agents are increasingly runs of hundreds or thousands: overnight batch jobs, autonomous research tasks, coding agents that work a ticket for six hours unattended. The cache framing has a blind spot at that scale, and it's worth naming precisely: the summaries still live in the window, and a window is a linear structure. Compress ten steps into one sentence and you've bought a constant-factor win, not an exemption from the shape of the problem.
Where compaction alone runs out
Say your compactor is good — genuinely good, keeping decisions and dead ends the way a careful summarizer should. Ten raw steps compress to one 50-token digest. That's a real 90%+ reduction per chunk. But every chunk still gets appended to the transcript, and every subsequent step still re-sends every prior chunk as prefill — the same quadratic mechanism as raw history, just with a much smaller constant and a much later onset.
SYS = 1500
OUT = 300
IN_COST, OUT_COST = 3.0 / 1e6, 15.0 / 1e6
def run_cost(N, digest_tokens):
total_in = total_out = 0
for k in range(1, N + 1):
# every step re-sends one digest per PRIOR chunk of 10 raw steps
prefill = SYS + ((k - 1) // 10) * digest_tokens
total_in += prefill
total_out += OUT
return total_in * IN_COST + total_out * OUT_COST
for N in (100, 500, 2000, 5000):
print(f"N={N:5d} ${run_cost(N, digest_tokens=50):.3f}")
N= 100 $0.968
N= 500 $6.338
N= 2000 $47.850
N= 5000 $232.125
That's with a good compactor — 50 tokens per ten steps, nothing wasted. The bill still climbs faster than the run length, because "compressed" is not the same as "gone." At 2,000 steps you're re-sending 200 digests every single call, on top of whatever's currently live. Push the run length another order of magnitude — which is exactly what autonomous, long-horizon agents are starting to do — and compaction alone stops being a fix and becomes a slower version of the same problem.
The structural difference: replay vs. lookup
Everything in the cache post shares one assumption: whatever the agent might need from the past should be sitting in the prompt when the model is called, because the model can only see what's in the prompt. That's true, but it doesn't mean every past digest needs to sit in every future prompt. Step 340 of a 2,000-step run almost never needs a fact from step 12 — and when it does, it needs one specific fact, not the accumulated shape of everything that came before it.
That reframes the problem from compaction to retrieval. Instead of asking "what do we keep in the window," ask "what does this step need, and where do we look it up." The transcript stops being something you replay in full (even compressed) and becomes something you query.
import re
from dataclasses import dataclass, field
@dataclass
class Entry:
step: int
kind: str # "decision" | "dead_end" | "fact" | "tool_result"
text: str
tags: set[str] = field(default_factory=set)
class TranscriptStore:
"""Append-only log of everything; the window never holds all of it."""
def __init__(self):
self.entries: list[Entry] = []
def append(self, step: int, kind: str, text: str) -> None:
tags = set(re.findall(r"[a-zA-Z_][a-zA-Z0-9_]{3,}", text.lower()))
self.entries.append(Entry(step, kind, text, tags))
def retrieve(self, query: str, k: int = 5) -> list[Entry]:
"""Cheap lexical retrieval: score by tag overlap, not embeddings."""
q_tags = set(re.findall(r"[a-zA-Z_][a-zA-Z0-9_]{3,}", query.lower()))
scored = sorted(
self.entries,
key=lambda e: len(e.tags & q_tags) + (0.1 * e.step), # tie-break: recency
reverse=True,
)
return [e for e in scored if e.tags & q_tags][:k]
That's deliberately not a vector database. Most agent transcripts have exactly the property full-text and tag-based retrieval is good at: entries are short, technical, and share vocabulary with the query that would need them (a step about "the serializer" retrieves other entries mentioning "serializer"). Reach for embeddings when recall on paraphrase actually matters for your workload — a support-ticket agent pulling from prior conversations, say — but don't install a vector store as a default. The store above is stdlib and runs in microseconds; that's the right cost for a lookup that happens every step.
Wiring it in, the per-step prompt changes shape:
store = TranscriptStore()
for step in range(max_steps):
upcoming_action = plan_next_action(state) # "call the serializer with region=eu-west"
relevant = store.retrieve(upcoming_action, k=5) # bounded by k, not by step count
prompt = build_prompt(
system=SYSTEM_PROMPT,
task=task_description,
working_state=state.render(), # small, always current
retrieved=relevant, # bounded, query-specific
)
response = model.call(prompt)
store.append(step, kind="decision", text=response.text)
result = run_tool(response.tool_call)
store.append(step, kind="tool_result", text=result)
The prompt at step 2,000 is the same shape and size as the prompt at step 20: system prompt, task, current state, and up to k retrieved entries. Run length stops being a term in the cost equation at all. That's a stronger property than anything eviction buys you — eviction keeps the window flat; retrieval keeps the per-step lookup flat regardless of how large the underlying log grows, because you're never asking the model to hold the whole log's worth of anything.
What retrieval costs you that eviction doesn't
This isn't strictly better, and claiming it is would repeat the mistake "Lost in the Middle" already taught — every mechanism that decides what the model sees can also decide wrong, and retrieval's failure mode is quieter than eviction's. When an evicted-and-summarized fact turns out to matter, at least a compressed trace of it is somewhere in the window. When a retrieval query misses, the fact isn't degraded — it's simply absent, and nothing in the response tells you it was needed. A recall failure looks identical to the model never having known the thing at all.
Two guardrails earn their cost:
Never retrieve the load-bearing facts — pin them. Constraints, the task definition, anything that must never silently drop belongs in working_state, always present, never subject to a query matching or missing. Retrieval is for the long tail of "did we already try this, and what happened" — not for anything the agent cannot afford to forget even once.
Log retrieval misses, not just hits. If you can, have the agent flag when it needed something from the past that its query didn't surface — a tool call that repeats an already-ruled-out approach is the retrieval-era version of the "stuck but busy" loop a bad compactor also causes. Without that signal, a systematically bad query function degrades every long run the same silent way a bad summarizer does, and you find out from the postmortem instead of the metrics.
When not to bother
For a run of 50 or even 150 steps, context-window-as-cache is simpler, cheaper to build, and sufficient — the summary-accumulation curve above doesn't bite until it's had hundreds of steps to compound. Standing up a TranscriptStore, writing a query function, and validating recall is real engineering effort that a short-lived agent doesn't need to pay for. The trigger isn't "my agent uses tools" or "my agent runs a while" — it's a run length where you've measured (not guessed) that accumulated digests are themselves a meaningful fraction of your token bill. Plot digest tokens re-sent per step against step number, the same way the quadratic post suggests plotting raw prefill. If that line is still flat at the length your agent actually runs, you don't have this problem yet.
The one-line version
Compaction makes the window smaller; retrieval makes the lookup stop scaling with run length at all — and past a few hundred steps, that's a different and stronger guarantee than a better summarizer can give you. Keep the load-bearing facts pinned, index everything else as an append-only log outside the prompt, query it per step with something as cheap as tag overlap before reaching for embeddings, and instrument for retrieval misses the same way you'd instrument for a bad compaction. The agents that need this aren't hypothetical — they're the ones already running long enough that this post's first cost table understates the bill they're paying today.
