The Model Reading My Benchmark Mattered More Than the Memory System Did

On 2026-08-11 I reran a benchmark I'd already published and changed exactly one thing: the model that reads the retrieved memory and writes the answer. Same memory system, same retrieved context, same judge, same rubric. YantrikDB's own score moved about 9 points on that swap.

Grounding, if you haven't heard of the thing being measured: YantrikDB is a persistent memory system for AI agents — it stores what happened in earlier sessions and hands the relevant parts back on demand. It makes no LLM call at ingest, which is where the 241-second ingest and 80ms retrieval numbers further down come from.

Two weeks before that rerun, Hindsight — a competing agent-memory system — had published a score on the same benchmark 25 points above mine. Nine of those 25 came from a model swap that has nothing to do with either memory architecture. More than a third of a published architectural gap, produced by changing the plumbing.

The comparison that got thrown out

The first version was the lazy one. BEAM-100K, an agent-memory benchmark, rubric-scored: YantrikDB at 0.611, Hindsight's published number at 0.862. My answerer was deepseek-v4-flash, theirs was gemini-3.1-pro-preview, and I filed that as a footnote.

Two adversarial reviews came back on it on 2026-08-11, one from a Codex-family model (gpt-5.6-sol) and one from Qwen (qwen3.8-max), both rejecting it for the same reason. An end-to-end agent-memory score has at least three things stacked inside it: how good retrieval is, whatever an ingest-time LLM does to the data on the way in, and how well the reading model uses what came back. YantrikDB and Hindsight differ in architecture. Hindsight runs an LLM fact-extraction pass at ingest and YantrikDB doesn't. The answerer models were different. Three variables moved together and I'd assigned all 25 points to the one I happen to build.

My first reaction was that this was reviewer pedantry — the reader model is plumbing, the memory system is the interesting part. Then I ran the swap to prove it, got the 9 points, and spent the next two days rewriting the paper instead.

Replaying their own context strings

Hindsight publishes, per query, the literal context string it injects into its answerer — one for a RAG configuration, one for a single-query configuration. Not a description of what its retrieval does; the string itself. So I took those strings unmodified and ran them through YantrikDB's own fixed answerer and judge: deepseek-v4-flash:0731 at temperature 0, the identical pair already scoring YantrikDB's own retrieved contexts. Nothing here calls Hindsight's live system; it's their published artifacts replayed through my scoring pipeline, so the only thing varying between rows is which memory system wrote the context.

Worth being blunt about one detail: the answerer and the judge here are the same model. deepseek-v4-flash:0731 writes the answer and then grades it, and self-grading has an obvious bias risk that I'm not going to dress up. What makes it usable is the frozen-context design — the same reader and the same judge process both systems' contexts under the same rubric, so whatever that model is inclined to reward, it rewards equally in both columns. The residual worry attaches to the absolute score level; the gap between the columns is what gets reported.

400 queries nested inside 20 conversations, all from BEAM-100K. All 400 query_ids matched across systems, so there's no partial-overlap story to explain this away.

The significance test needs a note, because the obvious test is the wrong one. Those 400 queries aren't 400 independent draws — they sit inside 20 conversations, and queries from the same conversation tend to succeed and fail together, so a per-query t-test would count 400 samples where there are really 20. The test therefore works at conversation level: flip the sign of each conversation's mean difference, enumerate every possible assignment, which with 20 conversations is 2^20 of them, a bit over a million, and p comes out at 0.438. The interval is a separate procedure — 4,000 conversation-clustered bootstrap resamples, giving [-0.021, +0.052]. A +0.015 rubric difference with an interval that comfortably straddles zero is a tie, and the paper's own word for it is "equivalent". Against Hindsight's single-query configuration the difference is +0.044, p=0.025, CI [+0.009, +0.077] — real, but single-query is the weaker of their two published configurations, so leading with it would be picking the easy target.

What did survive is cheaper than a win. YantrikDB reached that equivalent rubric score on 13,673 tokens of context against Hindsight's 23,689 — 42% fewer. The paper is careful here and I want to stay careful: part of that 42% is the cost of not carrying event time. Hindsight's contexts carry a median of 117 dates each. YantrikDB's carry a median of 5. "Not a free efficiency" is the paper's phrase, and it's the right call: that's a difference in what the two memory layers choose to emit. YantrikDB's two weakest categories in this eval are event ordering at 0.298 and temporal reasoning at 0.425. Those numbers and the token saving are one fact seen from two sides: a context with fewer dates in it is cheaper to read and worse at time.

Ingest and retrieval move the same direction, for a design reason and not a tuning one:

No LLM call at ingest is the choice; 80ms against 2,565ms is what the choice buys.

The part worth stealing

Back to the 9 points, the finding I'd defend hardest. Holding retrieval completely fixed and changing only the answerer moved YantrikDB's own score by more than a third of the entire published gap between two different memory systems. The paper puts it flatly: "The published gap is therefore overwhelmingly a property of the reading model, not of the memory systems." Any end-to-end "our agent memory beats X" number that doesn't name its answerer model and its judge model isn't a comparable number, and may be describing nothing more interesting than a stronger LLM reading the same context better. That's a complaint about how the category reports results, not about Hindsight, whose habit of publishing raw context strings is the only reason this experiment was possible.

A different reader, a different answer

There's a later experiment in the same paper, and leaving it out here would be editing rather than summarising. On 2026-08-15 I ran the rag-mode 2×2 again under an equalized frontier-class reader: moonshotai/kimi-k2.6 answering, meta-llama/llama-4-maverick judging — a different reader/judge pair from the deepseek one above. On 393 paired queries, YantrikDB scored 0.6331 rubric against Hindsight RAG's 0.5982: +3.5 percentage points, bootstrap 95% CI [+0.7, +6.2]pp, excluding zero.

That doesn't overturn the tie, and I'd rather be tedious about it than let anyone read it that way. A different reader model is a different question rather than a rerun of the same one, and the RAG-vs-RAG comparison under the shared deepseek reader stays the honest headline: equivalent. What the frontier run supports is narrower — under one stronger reader, on a slightly smaller paired set, a small edge that clears its own confidence interval. It is not the sentence "YantrikDB beats Hindsight," and writing that sentence would be doing the exact thing the previous section complains about.

Wrong twice, and what's still open

The reader-model confound was the big retraction. There's a second one, smaller, the same shape. I'd had a hypothesis that Hindsight's ingest-time extraction step was quietly discarding evidence later questions needed — an information-bottleneck story, and the one result that would have made YantrikDB structurally better instead of just cheaper. Category by category against their single-query configuration, it looked real. Against their RAG configuration it mostly evaporated. The paper's line: "We are withdrawing the information-bottleneck claim as unsupported by this experiment." That paragraph took longer to write than the whole results section.

Smaller things, in no order. The results page briefly said engine version 0.14.0; the version actually installed in the benchmark's virtualenv, per dist-info at 21:05 on 2026-08-11, was 0.13.4. That number turned out to matter slightly — a 2026-08-16 rerun of the same comparison on engine 0.15.0 scored 72.5% / 0.6375 where 0.13.4 scored 71.5% / 0.6107, a +0.027 rubric delta, bootstrap CI [+0.007, +0.045], small and clear of zero. The deepseek 2×2 is incomplete: YantrikDB's contexts were never run through Hindsight's answerer, so three of the four cells exist. Context length was never controlled either, so the token-efficiency and accuracy-equivalence findings stay two claims sitting next to each other rather than one. The local harness is a fork of vectorize-io/agent-memory-benchmark, which carries no license and can't be redistributed, so the paper's "Reproduce" link points at the upstream repo.

The paper went up 2026-08-12 at yantrikdb.com/papers/beam-frozen-context/ and until this post it hadn't been anywhere but its own domain. Take the boring thing from it: when you publish agent-memory numbers, name the model that read the context and name the model that graded the answer. Without those two names the number doesn't mean much, including when the person publishing it is me.

Pranab Sarkar, Independent Researcher