Textbook: Loop Robustness — Failures, Recovery, and Takeover

Source version of CodeSmith: v0.5.0 (commit 3a74c82f). All paths are relative to the repo root; line numbers refer to this version.
Intended audience: readers who have run an Agent loop in production and been tortured by dropped streams and silent stalls.

The two Tool Internals installments were about the inner strength of individual weapons; the Injection Defense installment stood watch over the entrances and exits of the context. This one climbs a level and takes on the loop itself. Article 3's anatomy diagram drew the Agent as a "think–act–observe" loop; what the diagram did not draw is that the loop has to turn for tens or hundreds of rounds, and every round must pass through four kinds of failure territory. Ablation studies warned long ago — a single missing tool-result feedback is enough to send an Agent into an infinite loop. More than half the gap between a production Agent and a demo is buried in the territory of failure handling; the textbook calls this territory failure and error recovery, and this installment aligns it with CodeSmith's source.

Taxonomy: Four Layers of Failure — Classify First, Then Count

The first step in engineering fault handling is not retrying; it is classification. By where they occur, faults fall into four layers:

The first judgment after a fault is caught is "is it worth retrying": retrying only makes sense for retryable errors (rate limits, overload, network jitter); non-retryable errors (invalid arguments, insufficient permissions) return the same result no matter how many times you replay them unchanged — a production-grade system maintains a mapping table from errors to recovery strategies, not a blanket "retry on error".

CodeSmith's version of this table is interesting, because the errors the engine sees are strings. The main conversation path goes through the rig adapter layer, and by the time an HTTP error reaches the engine it has already been stringified — so the engine layer uses a text-heuristic classifier (crates/agent-runtime/src/error_taxonomy.rs:327):

pub fn classify_error_message(message: &str) -> ErrorCategory {
    let lower = message.to_lowercase();

    if lower.contains("maximum context length")
        || lower.contains("context length")
        // ...... (context_length / prompt is too long / context window, etc.)
    {
        return ErrorCategory::InvalidInput;
    }
    if lower.contains("rate limit")
        || lower.contains("too many requests")
        || contains_status_code(&lower, "429")
        || lower.contains("quota")
    {
        return ErrorCategory::RateLimit;
    }
    if lower.contains("timeout") || lower.contains("timed out") {
        return ErrorCategory::Timeout;
    }
    // ...... (auth / permission / network / state / internal)
}

Plain-English translation: lowercase the error text and sort it by keyword into eight categories. It looks crude, but it holds to an important philosophy — the classifier would rather stay a bit coarse than misread a transient failure as deterministic.

There is a real lesson here: when reqwest reports a 503 it appends "(503 Service Unavailable)", and the word "unavailable" lands in the State category rather than Network. The fix was not to hone the classifier finer but to rebuild the retry arm as whitelist exclusion — only identity and authorization errors (auth/authz) and over-length context (which has its own dedicated recovery channel) hard-fail; everything else stays retryable.

The comment does the arithmetic in broad daylight (engine/turn/stream.rs:985):

Auth failures are deterministic — every retry re-fails identically — so surface them now. Everything else (...) stays retryable: the text classifier is heuristic, and a wrongly-deterministic verdict would hard-fail a transient failure after a single attempt — the bounded retry budget (3) is the cheaper mistake.

Misjudging a transient fault as deterministic and hard-failing it on the first try costs more than burning a few extra rounds of a bounded budget.

Beyond single errors, patterns need detecting too. Repeated-call fingerprints and consecutive-failure counters were covered by Article 10's loop guard (the third call of the same tool with the same arguments is blocked outright; three consecutive failures draw a warning, eight a halt) — no need to rehash. What this adds is the other class of fault, the kind that never shows up as an error — see the next section.

The Pipe Is Connected but No Water Comes Out: the Watchdog

The most dangerous failure mode of a streaming connection is not disconnection — disconnection raises an error immediately — but the silent stall: the connection is established, yet the data stops flowing, a water pipe that is connected but delivers nothing. SDK timeout mechanisms usually cover only connection establishment, not the transfer; a total-duration budget is wrong too, because a legitimate long think can go tens of seconds without emitting a single word. The right thing to detect is the silence gap: apply a separate deadline to every stream.next() (engine/turn/stream.rs:437):

loop {
    let item = match idle_timeout {
        Some(idle) => match tokio::time::timeout(idle, stream.next()).await {
            Ok(item) => item,
            Err(_elapsed) => {
                let error = format!(
                    "stream idle timeout: no events for {:.1}s, connection may have silently stalled",
                    idle.as_secs_f64()
                );
                if any_content_received {
                    // content already received → return Partial, present what was received to the user
                    ...
                }
                return StreamReduceOutcome::Empty { error };
            }
        },
        None => stream.next().await,
    };

Plain-English translation: don't clock the stream as a whole — clock each individual wait for the next event. A stream that keeps producing never trips it; only a genuinely mute one does. And what happens when it fires ties back into Article 4's billing scale: not one byte received (Empty) → transparently retry the entire request; some content already received (Partial) → present the partial content and do not retry, to prevent double billing.

It generalizes into a principle: every long-lived connection needs an independent liveness signal, not faith in the connection timeout — HTTP/2 keepalive (#103) and the consecutive-error cap of MAX_STREAM_ERRORS_BEFORE_FAIL = 5 are both variants of the same principle.

This watchdog also carries an "adaptive" touch of craft (engine_config.rs:22-32):

/// Default stream idle watchdog budget (P0-1). Long enough that a reasoning
/// model's quiet thinking phase (no text deltas yet, connection healthy)
/// doesn't trip it, short enough that a silently-stalled stream recovers
/// within a couple of minutes via transparent retry.
pub const DEFAULT_STREAM_IDLE_TIMEOUT_SECS: u64 = 120;
/// Default per-retry widening of the stream idle watchdog window (P0-1
/// adaptive retry). A provider silence gap longer than the base window would
/// kill every fixed-window retry in the same silent stretch; widening the
/// window per retry gives each attempt a longer lease.
pub const DEFAULT_STREAM_IDLE_RETRY_INCREMENT_SECS: u64 = 30;

On retry the window widens as base + increment × retries already spent (effective_stream_idle_timeout, turn/stream.rs:126) — a base window of 120 seconds, 150 on the first retry, 210 on the third. Without the widening, if the provider's silent stretch is itself three minutes long, all three retries would dash themselves against the same wall three times over. This is muscle that came home with the landing of improvement plan P0-1 (Article 4 mentioned it in one sentence; this installment completes the principle).

The Ladder of Recovery: Never Escalate When a Lower Rung Would Do

The textbook grades recovery means by how transparent they are to the user: silent retry → degradation and continuation → exposure to the user. The principle: the boundary of error handling is not the single request but the entire recovery loop — until recovery is confirmed impossible, intermediate errors should not surface to the consumer; only when every automatic means has been exhausted is the error presented, together with the actions already attempted.

Rung one: silent retry. The HTTP layer's general retry configuration looks like this (crates/agent/src/llm_client/mod.rs:407):

max_retries: 3,
initial_delay: 1.0,
max_delay: 60.0,
exponential_base: 2.0,
jitter: true,
jitter_factor: 0.1,
respect_retry_after: true,
retryable_status_codes: vec![429, 500, 502, 503, 504],

Exponential backoff (1s→2s→4s, capped at 60s) layered with ±10% random jitter — the comment says outright that the enemy is the thundering herd (massed clients retrying in sync and breeding secondary congestion). The jitter's source of randomness is the entropy of a UUID v4 — the style of a project that declines to pull in a rand dependency. respect_retry_after defers first to the server's wait hint (capped at one day).

The engine streaming layer's transparent retry has no backoff (Article 4 covered it: it retries only empty streams with zero content); only rate-limit errors sleep a fixed 2 seconds — Retry-After is already a string by the time it reaches the engine, so it is simply hardcoded.

The foreground/background distinction is this rung's hidden exam question: requests in the main loop must be retried, but retries of auxiliary calls like title generation and summary compaction would crowd the main path's quota and breed "retry amplification". CodeSmith's background compaction summary keeps its own shorter account (compaction/compact.rs): three retries, 1s/2s/4s backoff, and it recognizes only the three transient categories Network/RateLimit/Timeout; three straight failures book one trip on the circuit breaker and abandon that particular compaction, while the turn continues as normal.

The circuit breaker (compaction/circuit_breaker.rs) trips after 3 consecutive failures, half-opens for one probe after 5 minutes, and yields to a manual /compact — what it guards against is the textbook's death spiral: the recovery logic itself calls the LLM, the LLM fails again, and the money burns in a chain. Article 11's rapid-refill detector is its brother from the same school, guarding against the other spiral, where "compaction succeeds but the work is wasted".

Rung two: degradation and continuation. When retrying does not work, change the request itself. The textbook's example is hitting the output cap (generation truncated mid-flight by a length limit) — first silently raise the cap and resend; if that is still not enough, have the model pick up from the break point.

CodeSmith's answer on this rung is not continuation but context surgery: an over-length context that the gateway reports only after the stream has opened feeds into the same recovery channel as the pre-stream one (turn/stream.rs:981), with a budget of 2 (MAX_CONTEXT_RECOVERY_ATTEMPTS) and a three-step waterfall — micro-compact to clear old tool results → forced full compaction (bypassing the cache-protection floor) → hard-trim the oldest messages, keeping a floor of 4. On success the retry budget resets to zero and this step resumes without replaying the work.

The terminus of degradation is the local fallback: on the emergency path, when LLM compaction fails too, degrade to a purely local hard trim (host_executor.rs:2553) — the LLM being unavailable does not mean there is no way out.

Rung three: exposure to the user. Only when every means is exhausted does the error surface. Tool-layer errors take a different road — they do not end the session; they turn the error into the model's input: a hallucinated call receives "tool does not exist", malformed arguments receive an error carrying constraint hints, and both flow back as ordinary tool results for the model to self-correct on its next round (Article 16's repair ladder is the preprocessing for that road). The more specific the fed-back error, the higher the self-correction success rate — "the more structured the feedback, the better," cashed in the fault domain.

Takeover: The Credentials of Thought Cannot Travel

The last kind of recovery is the most extreme: the primary model stays unavailable, and the half-finished trajectory is handed to another vendor's model to keep running. The surface obstacle is divergent APIs; the real obstacle hides in thinking — tool calls and results differ in structure from vendor to vendor but agree in semantics, so re-rendering is enough; a model's thinking, though, is made of two parts: the readable text, and the credential the vendor attaches to the text (proof that this stretch of thinking really came from the model itself). Text, handed to another vendor's model, still reads; a credential, the moment it changes vendors, is void. Vendors verify with varying strictness — some never check, some reject every foreign credential — so a takeover scheme can only be designed against the strictest end.

Hence a textbook-grade design principle: trajectories should not be stored in any one vendor's API format, but as a neutral one — thinking is split into portable text and non-portable credentials, and on a switch the credentials are always discarded.

On this count CodeSmith scores "right by crudeness". The thinking block in the message history carries a single plain-text field (crates/agent/src/models.rs:143):

#[serde(rename = "thinking")]
Thinking { thinking: String },

No signature, no redacted_thinking — nowhere in the entire repository is there any logic that stores or retrieves a vendor's thinking credentials. The history stores its own neutral structure, and vendor differences are pushed to render time: the rig adapter layer's shaper decides, per target vendor, whether to strip or to replay (crates/providers/src/rig_adapter/shaper.rs:160):

A DeepSeek thinking-mode tool-call round must carry reasoning_content or the request 400s, so the shaper pads in the placeholder "(reasoning omitted)"; the vendors that reject the field get it stripped wholesale. Neutral in storage, dialectal at render time — the foundation for takeover is, without anyone quite intending it, the textbook posture.

This section's honesty checklist, viewed from another angle: nowhere in the repository is there a mechanism for "automatically switching to a backup model after the primary fails repeatedly", nor breakpoint continuation on finish_reason=length — in v0.3.0, stop_reason was carried out of the stream and bound as _stop_reason, never consumed; v0.5.0 put it to work, and not for continuation but for rejection: in rounds that end on length, tool calls that fail to parse are uniformly refused execution, and the resend request is fed back to the model (Article 22's termination proof and truncation rejection). Output-cap overruns are still handled by preventive budgeting (a per-turn output cap of 262,144 tokens, leaving headroom for interleaved thinking), not by after-the-fact remedy.

CodeSmith's recovery philosophy is transparent retry plus context surgery within one model, not failover between models — one model, one ledger, with the "swap the brain" variable kept entirely outside the system.

Summary: Every Layer of Recovery Keeps Its Own Ledger

Gathering this installment up:

1. Classify faults before counting them: four fault layers, an error→strategy mapping table, and a classifier that would rather stay coarse than rule a transient failure deterministic;
2. Silent stalls are caught by liveness signals: a per-event watchdog with a 120-second base window, widened by 30 seconds per retry;
3. Recovery escalates rung by rung: silent retry (backoff + jitter, separate accounts for foreground and background) → degradation and continuation (context surgery's three-step waterfall) → exposure to the user; tool errors flow back as model input;
4. Recovery logic that calls an LLM must have a circuit breaker, or the recovery itself becomes the new source of faults;
5. The foundation of cross-vendor takeover is a neutral trajectory format: thinking's credentials cannot travel, its text can;
6. The three "absences" are design too: no backup-model switching, no truncated continuation, no thinking credentials — complexity kept outside the system.

The trade-offs, laid on the table: retry budgets are bounded everywhere (3 retries, 2 attempts, 5 minutes), which means long failures must eventually surface as user-visible errors — a deliberate exchange of "silent money-burning" for "explicit failure"; a text-heuristic classifier means the occasional misclassification — and whitelist exclusion presses the cost of a misclassification down to "a couple more tries".

With this, the loop's skeleton is complete: Article 3 drew the anatomy, Articles 16 and 17 forged the weapons, Articles 19 and 20 blocked the adversaries hiding inside the content, and this installment fitted the loop with a fuse — v0.5.0 added one more bone that the skeleton diagram of the time did not yet have: the termination proof and truncation rejection for streaming output beheaded at the token cap (Article 22). The bones are all in; one last question remains: after this harness has done ten thousand jobs, will it be more practiced than it was on day one?