Part 2: We Fixed the Root Causes of Our AI Agent Incident. Routing Models by Step Cut Latency 86%

Note: this is a composite scenario built from common production failure patterns in agent systems. Ticket IDs, dollar amounts, latencies, and probabilities are illustrative. I drafted it with AI assistance and edited it myself.

In part 1, our billing agent settled a $48,000 credit it should have routed to a human. We showed why re-running the agent five times proved nothing, then used cut-point replay and a deterministic gate to prove the fix.

The gate stopped the bleeding. It did not explain why the agent reached the wrong conclusion in the first place. This post is about the upstream causes, and about the second half of the TICKET-8841 trace: why a six-step ticket took 42.6 seconds and cost $0.184.

Act III: Fixing the upstream root cause

The gate sits at Boundary 5. The agent got there because of what happened at Boundaries 2 and 3. When we held the post-mortem, someone immediately asked:

"Why are we still maintaining a RAG pipeline? Frontier models have 1M-token context windows now. Let's dump the entire 350-page Finance & Contract Handbook into the prompt."

It's a fair question, so we tested it.

Why "just stuff the handbook in" didn't win

Cost. A 350-page handbook is roughly 175,000 tokens, not 1M. Using an illustrative $3.00 per 1M input tokens:

175,000 tokens × $3.00 / 1M       = $0.525 per call
× 45,000 tickets/month            ≈ $23,600/month (uncached, one call per ticket)
with ~90% prompt-cache discount   ≈ $2,400/month

Our retrieval pipeline cost about $600/month. With caching, the cost gap is real but not dramatic, so cost alone isn't the strongest argument. (Plug in your own rates and call counts. Every extra step that carries the handbook multiplies these numbers.)

Recall. Long context is good now, but not perfect. Anthropic's launch numbers for Opus 4.6 on the 8-needle, 1M-token variant of MRCR v2 were around 76%. That's a synthetic benchmark at a much longer length than our handbook, so it says nothing precise about our documents. What it does say is that "the model will find the clause" is a claim you have to measure, and a miss on a financial contract is expensive.

Traceability. This is the argument that mattered most to us. With retrieval, the exact chunks that entered the prompt are a small, loggable, replayable artifact (exactly what a cut-point envelope wants). With a 175K-token dump, you can't tell which clause the model leaned on, and every edit to the handbook changes your entire prompt.

So we kept RAG. Our problem was that we were running 2023-era "RAG v1" in 2026.

Failure 1: Fixed-size chunking severed the condition from the formula

Our ingestion script split PDFs every 512 tokens. At 2:14 AM, the vector store held this:

chunk_041 (ended at token 512): "...For Enterprise Annual plans, seat reductions must be submitted in writing at least 30 calendar days prior to the renewal date. Mid-term or late seat reductions are non-refundable and apply only to the following renewal term."

chunk_042 (started at token 513): "Table 4.2 — Seat Downgrade Pro-Rata Credit Formula: credit_usd = (prior_seats - new_seats) * annual_seat_price * (days_remaining / 365)..."

The customer's message mentioned "downgraded 40 seats" and "billed", which matched the formula table well. chunk_042 scored 0.86 cosine similarity; chunk_041, the condition that made the formula inapplicable, scored 0.71 and missed the top-3 cutoff. The planner saw a formula with no precondition.

We made three changes:

1. Section-aware parent-child chunking. We index small child chunks for search but always return the full parent section (Section 4.2 and Table 4.2 together). Each document version is stamped with effective_from and superseded_at, so a stale policy can't be retrieved.
2. Hybrid search plus reranking. We pull the top 20 candidates using Reciprocal Rank Fusion over dense embeddings and BM25 keyword search, then pass them through a cross-encoder reranker to pick the top 5. On our 200-ticket regression suite, policy-exception recall went from 64% to 97%. (If you tune and report on the same suite, hold some tickets out.)
3. A regression test pinned to the incident. We reuse the harness from part 1, freeze the exact bad query rewrite from 2:14 AM, and check that the new retrieval still finds the notice clause:

def test_frozen_bad_rewrite_still_retrieves_notice_clause():
    session = CutPointSession(TRACE, mode="replay")
    bad_query = session.envelopes[("query_rewrite", 1)].output  # "...credit calculation table" (no "annual")

    chunks = retrieve(bad_query)  # new hybrid + rerank + parent-section pipeline

    assert any("30 calendar days" in c.text for c in chunks)

The upstream condition, a sampled rewrite that drops "annual", is still there. The pipeline is now robust to it.

Failure 2: Raw SQL over metrics with ambiguous meaning

At Boundary 3, the agent wrote SELECT active_seats FROM subscriptions against the warehouse through MCP. The query was valid. The problem was semantics: in our warehouse, active_seats is a live operational telemetry gauge, while contracted_billable_seats is the governed financial metric. The model picked a plausible column, and the nightly dbt job was mid-refresh, so it read 80 instead of 120.

Text-to-SQL was never the hard part; text-to-the-right-SQL is a semantics problem.

So we removed the run_sql() tool. The MCP server now exposes one governed tool that returns defined metrics in a single atomic read:

{
  "account_id": "acct_99201",
  "contracted_billable_seats": 120,
  "telemetry_active_seats": 80,
  "notice_window_days_remaining": -13
}

-13 is the whole story: the downgrade was requested 17 days before renewal, against a 30-day requirement. The agent no longer chooses between active_seats and contracted_billable_seats. The contract chooses, and the data comes from a consistent snapshot rather than a table mid-refresh.

With the full parent section and the governed metrics, the planner can't reach "automated overcharge error." And because the gate from part 1 sits at the exit, even a wrong planner decision can't move money.

Act IV: The latency and cost autopsy

Once the replay tests were green, we looked at why TICKET-8841 took 42.6 seconds and cost $0.184. The answer was embarrassing: we'd routed every boundary through a frontier reasoning model.

Asking a frontier model to rewrite a search query or format a JSON payload is paying for deliberation on a task that needs none. The planner decides whether the run succeeds, and the worker tier is close to interchangeable once the spec is unambiguous.

So we kept a frontier model on Boundary 4, the only step that needs multi-clause contract judgment, and routed the rest by task. I'm using model tiers instead of product names, because the rankings change monthly. Benchmark your own candidates against your own replay suite.

End to end, latency fell 86% (42.6s to 5.86s) and cost per ticket fell 82.7%. At 45,000 tickets a month, that's about $8,280 versus $1,430 in inference, before the hosting cost of any self-hosted models.

Two things stand out in the table:

• Correctness cost a little. Retrieval got slower and pricier (0.9s to 1.15s) because hybrid search and reranking do more work. That's the right trade.
• The gate made routing safe. The "0 false approvals across 200 replay traces" result isn't something we get from the cheaper models behaving well. For the high-value path it's structural, because the gate blocks above $5,000 no matter what any model says. Routing experiments can't regress it, and accuracy parity on the rest of the suite is the thing we measured.

The pattern

Across both posts, the same shape keeps showing up. Probabilistic models are extraordinary at turning messy human intent into structured plans. They're bad at being their own database catalog, access-control layer, and regression suite. What worked was wrapping them in deterministic engineering:

1. Semantic contracts going in: parent-section retrieval and governed metric tools instead of raw SQL.
2. Surgical compute in the middle: one frontier call for the judgment step, cheap and fast workers elsewhere.
3. Hard invariants at the exit: money-movement gates in code, outside the agent.
4. Frozen cut-point replay in CI: so a 2:14 AM incident becomes a 14ms deterministic test forever.

If you're debugging an agent right now, try this first: pick your last real incident and ask whether your fix has ever been run against the exact upstream state that caused it. If the answer is "I re-ran it a few times," start there.