94.7% on LoCoMo — and why most of the gap between published memory numbers isn't the memory

94.7% on LoCoMo at 5.0K context tokens — 95.0% when re-graded by Mem0's own judge. The
more useful finding is what we measured on the way there.

The number, with its configuration attached

94.7% on LoCoMo, all 1540 questions, at 5.0K context tokens per question. It is
the highest LoCoMo number we are aware of, published or otherwise, and we can hand you the
code and the per-question outputs for it
(github.com/todoforai/livemem).

The nearest published row is Mem0's April 2026 algorithm at
92.5, delivered in 6,956 mean context tokens;
then MemMachine's 91.7% for v0.2. So: +2.2 points on about 30% less context.

Per category: open-domain 97.0%, single-hop 93.6%, temporal 93.5%, multi-hop 81.2%.
That last one is where the remaining work is, and we'd rather print it than average it
away.

How much of that is the grader? This is the first question we would ask, so we
answered it: we re-graded the identical 1540 answers with gpt-4o-mini running Mem0's own
ACCURACY_PROMPT — the judge model and prompt behind their published LoCoMo numbers — and
got 95.0% (+18 flips to correct, −13 the other way). The two judges agree within 0.3
points on these answers. The lead is not a grading artefact.

How much of it is the ensemble? The answers come from two passes over the same
retrieved memory: gemini-flash and claude-haiku-4-5 answer the identical 5.0K block
independently, then a rule hierarchy (agree > commit > abstain) settles the 82
disagreements. One pass alone scores 92.4%. So +2.3 points are answer-side, bought with
a second model call and ~6 seconds per question instead of one — not with more context.

That distinction turns out to be the theme of this entire post.

And before anyone else says it: LoCoMo at 94.7% is a nearly saturated benchmark. Mem0
say so themselves
— "older benchmarks like LoCoMo are less informative today," because they don't stress
conflicting updates, identity ambiguity or long-running workflows. We agree, and we still
publish the row, because a saturated benchmark is exactly the regime where the unreported
variables — answerer, judge, context budget — decide the ranking. Two points of headroom
and a seven-point answerer effect is not a leaderboard; it's a measurement problem. The
rest of this post is that measurement problem.

The setup

We build TODOforAI, an agent that runs long-lived tasks on your
machine. Long-lived means it has to remember: what you decided three weeks ago, what your
deploy procedure is, which of two conflicting instructions is the newer one.

So we built a memory system, and then we did the thing you're supposed to do: benchmarked
it against published numbers instead of trusting our own demos.

Two public datasets:

• LongMemEval_S — 500 questions over long, multi-session chat histories. Single-session recall, multi-session aggregation, temporal reasoning, knowledge updates.
• LoCoMo — 1540 questions, heavier on temporal and multi-hop reasoning.

Both run through AMB, an
independent harness that fixes the dataset, the answering model and the judge for every
memory provider it runs.

LoCoMo went well from early on. LongMemEval was the humbling one, and still is: our
first honest full-dataset number was 85.4%, against Mem0's published 94.4%. We are
at 87.8% today — better, still behind, and behind at 4.2K context tokens against their
6.8K, which is the only part of that comparison we're happy with.

That 9-point hole is what sent us looking for bugs, and the bugs turned out to be more
interesting than the number. It is also why the rest of this post spends more time on
LongMemEval than on the benchmark we lead.

Chasing it

The obvious hypothesis is that their architecture is better. So we went looking for what
we were doing wrong, and we found real things:

Deduplication was deleting evidence. A question like "how many plants did I buy last
month?" needs every instance, not the top-k most similar ones. Our dedup merged "bought
2 basil plants" and "bought 3 basil plants" — near-identical sentences — silently
destroying an instance the count depended on. Making dedup number-aware (two facts with
different digits are never duplicates, however similar the prose) was worth +4.3 points
on a 94-question validation subset (78.7% → 83.0%).

Extraction was too tasteful. Our prompt skipped things "an LLM already knows." Then a
question asked "what was that Borges quote you gave me?" — public knowledge, yes, but
the specific thing we said in that conversation is not in any model's weights. The
lesson generalizes: extraction should be greedy, because a fact never extracted can never
be retrieved. Selectivity belongs to the selector, not the extractor. This fixed three
questions on our subset that had been unanswerable.

Some answers don't live in facts at all. They live in a conversation turn that no
summary preserved. Adding a retrieval configuration that mixes entity cards with 3-turn
conversation-window excerpts alongside the distilled facts took the full 500-question run
from 85.4% to 87.8%.

Real improvements, real points. Still not 94.4%.

The variable we hadn't controlled

Here's what we should have checked on day one. We had been answering with the cheapest
model we could — gemini-flash-lite — deliberately, for the reason above.

Mem0's published harness config uses gpt-5 as the answerer and gpt-5 as the judge.

So we ran the control: same memory system, same extracted facts, same retrieval, same
rendered context. Swap only the answering model.

+7.4 points from the answerer alone. And the judge is a free variable too — though
note the stronger judge was stricter, not more generous: it accepted two hedged answers
we'd been denied and rejected four we'd been given.

Be careful what this does and doesn't show. It does not prove Mem0's 94.4% is "really"
lower, or that the gap between two systems is entirely answerer choice — we never ran
their system on our harness. What it shows is narrower and still useful: on the same
memory, configuration differences of the size that separate published rows are worth
about as many points as the memory architecture itself. Our own comparable-ish number
is 90.4%, still short of 94.4%. We just no longer believe the raw distance between two
differently-configured rows is a measurement of anything.

What we actually learned

1. Cross-harness memory numbers are not a ranking. The clearest evidence isn't even
ours: in the published literature Mem0 appears at 94.4% (self-reported) and at
67.6% (measured in a third-party paper). Same system, 27 points apart, because the
harness, answerer, judge prompt and budget differ. Any table mixing sources — including
the one in our repo — should be read as a
collection of separately-configured experiments, not a leaderboard.

2. Judges are worth points, not decimals — and not only via the prompt. Our
single-pass answers scored 92.4% under claude-haiku-4-5 and 95.6% under gpt-4o-mini,
on near-identical judge prompts. Nothing about the memory changed between those two
numbers; only which model read the same grading instruction. (On the ensemble answers the
same two judges land within 0.3 points of each other — the size of the judge effect is
itself configuration-dependent, which is the point.)

3. Context size is a hidden axis. We run at ~4–5k tokens per question, measured after
rendering. Mem0's new algorithm reports its 92.5 at 7.0K and its 94.4 at 6.8K; the
Hindsight run on the AMB leaderboard reports 43.6k context tokens per question — about 9×
ours. Accuracy per token is a different ranking than accuracy, and only one of the two is
usually reported. Credit where due: Mem0 do report mean tokens next to every score, which
is more than most leaderboard rows offer.

4. Our own most embarrassing finding was in our token accounting. Our "5000-token"
budget was actually delivering 5742 real tokens. Date prefixes (- [2026-08-14]) are
15 characters but 10 tokens — dates are token-dense — and across a full block that
came to roughly 1.6k tokens we never charged for. After enforcing the real budget, delivered
context dropped to 4182 tokens and the score on our subset dropped from 83.0% to 77.7%. A
rerun with the budget raised so delivered context matched the old 5742 recovered to
81.9% — so the drop was the missing context, not lost capability. The uncounted tokens
had been buying real answers. If you don't measure delivered tokens, your budget is
decoration.

5. The dominant remaining error family is aggregation. "How many X in total", "what
percentage of Y". The instances are usually in the retrieved context — the answerer just
doesn't count them reliably. That's not fixed by retrieving harder. It's also a direct
cost of our weak-answerer choice: flash-lite stresses retrieval honestly, but it fails
arithmetic that a stronger model wouldn't. There's no free methodology.

Where LoCoMo stops being enough

Mem0's benchmark post is worth reading precisely because it argues against its own top
row, and its list of failure modes matches ours: paraphrase overfitting (systems pass
when the query echoes the stored wording), temporal revision (the user changed their
mind and both facts are still in the store), identity ambiguity, mixed-granularity
scoping (per-user vs per-project vs per-device), and scale — benchmarks run over
hundreds of events, production over millions.

Their own numbers show the drop: 92.5 on LoCoMo and 94.4 on LongMemEval_S, but 64.1 on
BEAM 1M and 48.6 on BEAM 10M — same system, same year, thirty to forty-five points lower
as soon as the benchmark contains interference, updates and long-horizon chains. That is a
much more honest picture of where memory actually is than any 9x.x% row, including ours.

We have not run BEAM yet. It's next, and we'll publish it the same way — with the answerer,
the judge, the delivered token count and the per-question outputs attached — whatever the
number turns out to be. Our expectation, stated in advance so it's falsifiable: our
multi-hop score of 81.2% is the honest predictor of how we do there, not our 94.7%.

The metrics they suggest reading instead of raw accuracy — recall@K, end-to-end task
success, context inflation, cross-session continuity, stability under paraphrase — are
the right list. We already publish one of them (delivered tokens per question, item 4 below
is what happens when you don't) and we're missing the other four. So is nearly everyone.

The code

We published the reference implementation: github.com/todoforai/livemem

The core is about 200 lines of TypeScript, because the architecture genuinely is small:

conversations ──extract──▶ dated facts + embeddings ──pack──▶ ≤ N-token block
                (1 LLM call,                          (cosine + greedy knapsack,
                 offline)                              no LLM on this path)

The bet the design makes: extract everything once, select per question, never call an
LLM to retrieve. Selection is cosine similarity plus a greedy knapsack under a token
budget — tens of milliseconds — so you can afford to re-select for every single question
instead of maintaining one static "user profile" that's stale by definition.

Our hosted API adds the tuned extraction prompting and the retrieval refinements described
above — number-aware dedup, entity cards, conversation-window units — and the benchmark
rows above were run in that configuration; the repo is the same architecture without that
tuning. The benchmark harness in bench/
runs against either one through the same provider, so you can measure the difference
yourself rather than taking our word for it.

The per-question outputs behind our principal rows — answers, judge verdicts,
context-token counts — are in that repo, each labelled with the answerer, judge and budget
it used. Some intermediate numbers in this post come from validation subsets we ran during
development and did not publish as full artifacts; they're marked as such. Numbers from
other systems are linked to their published source and were not re-run by us. If we got
something wrong, it's falsifiable, which is the only claim worth making about a benchmark.

We know where the next points are: multi-hop at 81.2%, aggregation, LongMemEval_S
where we still trail Mem0 by 6.6 points, BEAM — which we haven't run and expect to hurt —
and the hosted live path still behind our offline one. We'll publish the next number the
same way — measured against this one, with the artifacts attached.

If you want the memory without running it yourself, it's what powers agent memory in
TODOforAI.