Recall@1 went from 13% to 50%. Recall@5 from 13% to 80%.
Over the same week, the number of questions my agent answered end-to-end went from 2 in 10, to 1 in 10, to 0 in 10.
Every component metric said I was winning. The product was dying.
The setup
A data analysis agent over a lakehouse of banking and economic time series. You ask a question in plain language, it finds the right series, builds a table, derives columns, runs change-point analysis, draws the chart.
The agent never sees the data. It sees a catalog, one row per measurable quantity, with a name, unit, frequency, coverage. It picks; the database computes.
The catalog has 55,532 entries. That number is the whole story.
The paradox
Semantic search had been off during a data migration. Retrieval was keyword matching, and keyword matching wants every token of your question to appear in the label. Ask for "inflation" and you get a balance-sheet adjustment line, because the consumer price index is called "Consumer Price Index" and the word "inflation" is nowhere in it.
before: recall@1 = 13% recall@5 = 13%
after: recall@1 = 50% recall@5 = 80%
Then I ran the end-to-end scenario. Before the retrieval work, 2 runs in 10 completed. After: 1 in 10. I assumed noise, improved retrieval once more, ran again: 0 in 10.
What the model was actually doing
Every failing run looked the same. Search the catalog. Search again. Twelve, seventeen, twenty-three searches in one turn — and never once call the tool that builds the table.
The obvious reading is indecision: at 55,532 options it cannot commit. I had three theories along those lines and started building a fix for the second one.
Then I turned on the reasoning trace and read what the model said to itself.
"The first search returned 8 candidates, but the output was empty (only '8 candidates, best to worst:' with no list). Let me try searching again with different terms."
"Strange. The search results are truncated."
It was not refusing to commit. It was looking for evidence that had been deleted out from under it.
The line
Context management had a rule. Keep the last three tool results verbatim, collapse older ones to their first line:
VERBATIM_RESULTS = 3
def _collapse(result: str) -> str:
return result.split("\n")[0]
Here is what a search result looks like:
8 candidates, best to worst:
IMF - GDP, Nominal (USD) · Türkiye (TUR) … [EVDS_TP.IMFGDPUSDN.TUR]
Consumer Price Index (General) … [EVDS_TP.FE.OKTG01]
… six more lines
The first line is the header. The rule kept the header and deleted the list.
The model was shown "8 candidates, best to worst:" with nothing after the colon, concluded the search had returned nothing, and searched again.
At ten searches in one turn:
total prompt 8,646 chars (~2,161 tokens)
tool results 30% of the window
candidate ids visible 18 of the 80 it had been shown (78% deleted)
The window was 30% full. The rule was destroying the turn's working memory to save 1,800 characters.
Why better retrieval made it worse
Better retrieval meant plausible candidates on the first search. Plausible candidates invite a second search to compare against. Every search past the third deleted another result.
- Bad retrieval → the model gives up early → few searches → few deletions
- Good retrieval → the model explores → many searches → the good candidates are destroyed
Improving one component degraded the system through a mechanism neither component owned. Recall@1 was honestly improving the entire time it was killing the product.
It turned out I had written the same mistake in two more places, an 800-character summary cap that silently cut lists at the fifth candidate, and a shortlist that filled oldest-first so it dropped whatever the newest search had just found. All three were correct for the catalog I had when I wrote them: thirteen entries. None were revisited when it grew four thousand times.
What fixed it
Not a smarter model, not a better prompt. Three changes about what the model is shown:
- a collapsed result keeps its content, not its label
- every search re-states the turn's accumulated shortlist, newest first
- results are grouped by catalog structure instead of returned as a flat ranked list
before after
runs completing all three turns 0/10 9/10
turns building a table ~7/30 29/30
discovery calls per turn 12 2
Retrieval quality did not change across that fix. Recall stayed at 50%/80%. Grouping changes presentation, not ranking — and presentation was the binding constraint all along.
"GDP, Nominal · Türkiye — one of 196 rows under this heading" is a decidable choice. The same line in a flat list is not.
The industry-standard fix that backfired
Faced with "the model doesn't know how a question of this shape is answered here", the textbook answer is a verified query repository: question-and-answer pairs injected as worked examples. Snowflake ships exactly this. I built it.
grouped results only 9/10 runs
+ verified query repository 3/10, then 5/10
It cost six runs in ten. Half the damage had a cause written in my own interface contract six weeks earlier:
The system prompt must not contain family lists or series ids. A model that has seen an id in its prompt will invent variations on it.
My examples carried the ids of the families they resolved to. I had read that file. I shipped it anyway.
The other half was simpler: with grouped results, the model could already see where matches clustered. An abstract example gave it something extra to reason about that it did not need. Snowflake needs the repository because its semantic model is capped at 32K tokens and it cannot show the catalog. I could. Two solutions to one problem, interfering.
Three things I'd tell myself a week earlier
Read what the model says to itself. Three sessions of theorising about indecision were settled by one run with the reasoning trace on. The model had been describing the bug in plain language the whole time.
Component metrics can rise while the system dies. Recall went up monotonically across the exact window where end-to-end went to zero.
Every constant has an invisible scale attached. VERBATIM_RESULTS = 3, an 800-char cap, oldest-first fill — all correct at thirteen entries, all silently wrong at 55,532. Grep your constants and ask what size they were written for.
