Stop dumping raw message arrays into LLMs and start using structured state with sliding windows.
The Bottleneck in Production
The most common mistake when deploying AI agents is treating chat history as an append-only log. In early prototypes, appending every user turn, tool response, and raw JSON blob directly into the messages array works fine.
In production, this pattern collapses after twenty turns. Token usage scales linearly with conversation depth, driving up API latency and inference costs. Worse, models experience "lost-in-the-middle" degradation, forgetting early constraints or crashing altogether due to token limit errors.
# The naive anti-pattern: unbounded list growth
messages.append({"role": "user", "content": user_input})
messages.append({"role": "assistant", "content": llm_response})
# 30 turns later: 15,000 tokens wasted on stale tool payloads
response = client.chat.completions.create(model="gpt-4o", messages=messages)
Dumping unpruned histories into your LLM turns your database into an expensive latency trap.
The System Architecture & Fix
The solution is decoupling ephemeral dialogue from persistent conversational state.
Instead of forcing the LLM to re-parse the entire conversation history on every turn to understand what happened ten minutes ago, we split context into two distinct layers:
- Sliding Window Buffer: Retain only the last $N$ turns to preserve immediate conversational rhythm and flow.
- Structured State JSON: Maintain a compact, deterministic key-value store of critical facts (e.g., active user goals, collected form values, resolved errors).
Incoming User Turn
│
▼
┌─────────────────────────────────────────┐
│ Context Assembler │
│ ─────────────────────────────────────── │
│ 1. Static System Prompt (Identity) │
│ 2. Current State JSON (Facts & Goals) │
│ 3. Sliding Window Buffer (Last N Turns) │
└─────────────────────────────────────────┘
│
▼
LLM Inference (Bounded & Predictable)
This ensures your token payload stays flat whether a session lasts 3 turns or 300 turns.
The Implementation
Here is a lightweight context manager you can drop directly into your backend service pipeline.
from typing import Any, Dict, List
def build_bounded_context(
system_prompt: str,
raw_history: List[Dict[str, str]],
state_payload: Dict[str, Any],
max_turns: int = 6
) -> List[Dict[str, str]]:
"""Assemble a token-bounded context payload with structured state."""
# Enforce strict sliding window on ephemeral chat history
trimmed_history = raw_history[-max_turns:] if len(raw_history) > max_turns else raw_history
# Inject current state directly as a system-level context injection
state_injection = {
"role": "system",
"content": f"CURRENT_SESSION_STATE: {state_payload}"
}
return [{"role": "system", "content": system_prompt}, state_injection] + trimmed_history
This pattern provides deterministic context bounds. Your backend guarantees that the context size passed to the provider never exceeds your calculated budget:
If an agent needs to update persistent state (like a shipping address or user intent), extract that state asynchronously or via tool calls, store it in your database, and inject the clean JSON dictionary on the next invocation.
Production Lessons & Takeaways
- State belongs in your database, not the context window: Treat the LLM context like RAM and your database like a hard drive. Extract facts into structured fields instead of making the model re-read 40 turns of history.
- Keep $N$ between 4 and 8 turns: In practice, immediate dialogue context rarely requires more than the last 3-4 round trips. Anything older is usually noise.
- Make state updates idempotent: Update your structured state schema using explicit tool/function calls rather than free-form LLM summaries to avoid hallucinated state drift.
