Code Agent Dissection (09): When the Conversation Gets Too Long, What Happens When Tokens Run Out?

Where the Problem Comes From

When a coding agent runs, every ReAct step appends messages to the history: user input, model response, tool calls, tool results. After twenty or thirty steps, the history might contain hundreds of messages and tens of thousands of tokens.

LLM context windows are finite (default 128k tokens). Sending the full history verbatim to the model has two problems:

1. Overflow errors: Exceeding the context window returns a PROMPT_TOO_LONG API error
2. Quality degradation: Even without overflow, overly long history dilutes recent information and scatters model attention

There are two solution approaches: truncation (discard old messages) and summarization (compress old messages into a summary). MyCodeAgent uses summarization, with one key design principle: history is never deleted; compression only affects the view given to the model.

Core Design: Separating History from Model View

This is the prerequisite for understanding the whole mechanism — get these two concepts straight first:

HistoryManager (complete fact log, append-only, never deleted)
    All original messages: user / assistant / tool / summary
    The 'truth', the source for transcript writes and crash recovery

         ↓  build_model_view() called once per step

Model View (bounded projection given to the LLM)
    System messages + subset of history (possibly compressed)
    The 'view', only affects this LLM call, doesn't modify history

The benefit of separation: compression is a reversible read-time operation. The original history remains complete and can be rebuilt at any time; crash recovery doesn't depend on compression state.

Overall Flow

At the start of each ReAct step, the loop calls _prepare_step_context(), which runs three sequential steps:

step N begins
    ↓
compact_if_needed()        check if compression is needed; trigger if so
    ↓
build_model_view()         project history into message list for this step
    ↓
llm.invoke_raw(messages, tools=...)   send to LLM

Compression and projection are two independent steps: when compression hasn't triggered, projection is the full history; after compression, projection is the summary plus the most recent N rounds of originals.

Step 1: Deciding Whether to Compress

ContextBudgetPolicy.should_compact() makes this decision:

# runtime/context/budget.py
def should_compact(self, *, messages, pending_input, last_usage_tokens):
    threshold = int(self.config.context_window * self.config.compression_threshold)
    #           default: 128000 × 0.8 = 102400 tokens

    # Two estimation sources; take the larger (most pessimistic, to avoid missing triggers)
    estimated_from_messages = self.estimate_tokens(messages, pending_input)
    estimated_from_usage = last_usage_tokens + len(pending_input) // 3
    estimated = max(estimated_from_messages, estimated_from_usage)

    if estimated >= threshold:
        return CompactDecision(True, "threshold_exceeded", ...)

estimate_tokens() approximates token count as character_count // 3 — not precise, but conservatively safe. Taking the max of two sources handles the case where the character estimate is too low: if the last LLM call actually consumed many tokens, use the actual usage.

Where context_window and compression_threshold come from:

# core/config.py (read from .env)
context_window: int = 128000          # CONTEXT_WINDOW env var
compression_threshold: float = 0.8   # COMPRESSION_THRESHOLD env var
min_retain_rounds: int = 10          # MIN_RETAIN_ROUNDS env var

The trigger threshold is context_window × compression_threshold — compression happens at 80%, not 100%, to leave room for the model's output tokens.

Step 2: Compression — Producing a Checkpoint

When compression triggers, ContextCompactor.compact() executes:

# runtime/context/compact.py
def compact(self, messages):
    # 1. segment into rounds by user message boundaries
    rounds = self.round_segmenter.identify(messages)
    #    rounds = [Round(0,5), Round(6,12), Round(13,18), ...]
    #    each Round starts at a user message and ends before the next user message

    # 2. retain the most recent min_retain_rounds rounds (default 10) as originals
    retain_start_round = len(rounds) - min_retain_rounds
    retain_start_idx = rounds[retain_start_round].start_idx
    messages_to_compact = messages[:retain_start_idx]   # the older portion

    # 3. pass old messages to summary_generator (LLM call) to produce summary
    summary = self.summary_generator(messages_to_compact)
    # summary_generator has timeout protection (default 120s); returns None on timeout, no compression

    # 4. create checkpoint: summary text + retain_start_idx
    checkpoint = self.compact_store.create_checkpoint(
        summary=summary,
        retain_start_idx=retain_start_idx,
        ...
    )

Key: compact() doesn't modify any messages in HistoryManager. It only stores a checkpoint (summary + split point) in CompactStore. Original messages remain intact.

RoundSegmenter's split logic is simple: every role="user" message starts a new round:

messages: [user₁][assistant][tool][tool][user₂][assistant][tool][user₃]...
rounds:   |────── Round 1 ────────|──── Round 2 ─────|── Round 3 ──...

The most recent 10 rounds are retained; everything before is sent for compression.

What is summary_generator? Created by create_summary_generator(llm, config) in factory.py, returning a closure. When called, it serializes the old messages to text, sends them to the LLM, and a prompt requests a structured summary (completed goals, key decisions, modified files, etc.).

Step 3: Projection — Deciding What the Model Sees

Every build_model_view() call invokes ProjectionBuilder.project():

# runtime/context/projection.py
def project(self, source_messages):
    checkpoint = self.compact_store.active_checkpoint

    if not checkpoint:
        # No compression yet: model sees full history
        return ProjectionResult(messages=source_messages, mode="full_history")

    # Checkpoint exists: fold old history
    summary_msg = Message(content=checkpoint.summary, role="summary")
    recent_messages = source_messages[checkpoint.retain_start_idx:]

    return ProjectionResult(
        messages=[summary_msg] + recent_messages,
        mode="compact_checkpoint",
    )

The message structure the model sees after compression:

[system messages × N]
[summary message]   ← "Previously: user requested refactoring of auth module; X, Y, Z completed..."
[most recent 10 rounds of originals]   ← complete user/assistant/tool messages

This is 'read-time projection': project() reads from the original source_messages every time, dynamically generating the view without modifying any data. The retain_start_idx in the checkpoint is the split point, telling projection 'from where to start retaining originals.'

Two Trigger Modes

Compression has two trigger moments for different scenarios:

Proactive (at the start of each step): compact_if_needed(), called in _prepare_step_context(). Triggers when estimate exceeds threshold — preventive compression.

Reactive (on PROMPT_TOO_LONG): reactive_compact(), in the inner while loop, triggered when an LLM call fails due to context being too long. After successful compression, rebuilds the model view and retries.

# Reactive trigger path (loop.py)
try:
    raw_response = llm.invoke_raw(messages, ...)
except Exception as exc:
    if classify_model_error(exc).kind is ModelErrorKind.PROMPT_TOO_LONG:
        compact_info = host.context_engine.reactive_compact(...)
        if compact_info.get("compacted"):
            # rebuild model view, inner continue to retry
            messages = host.context_engine.build_model_view(...).messages
            continue

What Happens When Compression Fails

When summary_generator fails (LLM timeout, network error), compact() returns {"compacted": False, "reason": "summary_unavailable"} without making any changes.

Proactive trigger: skip compression; this step continues with uncompressed history. The next step re-evaluates whether to compress.

Reactive trigger (PROMPT_TOO_LONG): compression fails → unrecoverable → loop terminates with an error message. This is the worst case; it rarely occurs in practice because proactive triggering usually intervenes first.

Design Highlights

History never deleted: HistoryManager is append-only; compression doesn't modify history; can be fully rebuilt from transcript after a crash.

Read-time projection rather than write-time truncation: Truncation is destructive (information lost); projection is reversible (checkpoint persists, can be discarded and reprojected anytime).

Two estimation sources take the max: Uses both character-count estimation (current message content) and previous round's actual token usage; taking the larger ensures conservative estimates don't miss triggers.

Compression timeout doesn't crash: summary_generator uses ThreadPoolExecutor internally for timeout control; if the LLM summarization call times out, it returns None directly without affecting the main loop — this compression is simply skipped.

Summary

About the Source Code

All analysis in this series is based on the open-source project MyCodeAgent.

The source code includes inline comments aligned with the series' explanation order at key locations — you can follow along while reading, or clone it, run it, modify it, and extend it to build your own agent.

git clone https://github.com/chendongqi/MyCodeAgent
cd MyCodeAgent
cp .env.example .env   # fill in your LLM API key
uv sync
uv run python main.py

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage