Memory layers for AI agents love to quote LongMemEval. You will see 94%, 95%, 96%. Those numbers are real, but they usually measure retrieval: was one of the right sessions in the top 5 results? (recall_any@5). LongMemEval's own metric is different: did the model answer the question correctly?, graded by a GPT-4o judge.
I wanted to know how far apart those two numbers are, so I measured both on the same questions.
The setup
agentmemory reports 95.2% R@5 on LongMemEval_S, and to its credit it says plainly that this is retrieval recall and not QA accuracy. It also publishes the sessions its hybrid search (BM25 + embeddings) returned for each of the 500 questions.
So I took exactly those results, unchanged, pinned by commit and hash:
- For every question, the top 5 sessions agentmemory returned, every turn in full.
- The same answer model (gpt-5-mini), the same answer prompt.
- The official LongMemEval judge (gpt-4o-2024-08-06).
Then I did the same with Graph-MIND, the memory layer I build, whose shipped retrieval returns a ~3.4k-token packet of individual turns.
Only the retrieval differs. Same questions, same reader, same judge.
The result
| answer accuracy (499 Qs) | never used for tuning (379) | text the reader sees per question | |
|---|---|---|---|
| agentmemory (its top 5 sessions) | 80.4% | 78.6% | ~16.5k tokens |
| Graph-MIND | 89.0% | 87.1% | ~3.4k tokens |
Graph-MIND alone was right on 64 questions and agentmemory alone on 21; exact McNemar p = 3.3e-6.
By question type (Graph-MIND / agentmemory):
- multi-session: 80.8 / 65.8
- temporal reasoning: 88.9 / 77.0
- single-session user: 98.4 / 87.5
- knowledge update: 94.4 / 90.3
- preference: 75.0 / 71.4
- abstention: 93.3 / 90.0
- single-session assistant: 94.6 / 96.4 (agentmemory wins this one)
Why 95% recall becomes 80% answers
Three things happen between "a right session is in the top 5" and "the answer is right":
- Any is not all. recall_any@5 counts a hit if one answer session is retrieved. Multi-session questions ("how many times did I...", "how long between...") need several. That is where the gap is widest: 15 points.
- More text is not more signal. Five whole sessions are ~66k characters. The answer is a sentence or two in there. The reader has to find it, and on temporal questions also work out which of several dated mentions is the current one.
- Rank is not position. A session ranked fifth still lands in the reader's context, but next to four sessions that look relevant and are not.
None of this makes agentmemory bad; 80% on LongMemEval with a small reader is respectable, and whole sessions are its natural unit. The point is that R@5 and QA accuracy are different measurements, and comparing one project's R@5 with another's QA (or with each other's R@5 on different units: sessions vs. turns vs. chunks) does not tell you which one answers better.
Caveats, because I build one of these
- I picked the reader and the judge setup; it is the one every Graph-MIND number uses, and it is the official judge.
- Graph-MIND was developed against 120 of these 500 questions. The untuned column (379 questions) is there for that reason; the gap holds.
- I used agentmemory's published retrieval file, not a fresh run of its current code.
- Whole sessions are agentmemory's best case here: its reader sees about five times more text.
Everything is reproducible: the script is research/rival_agentmemory.py, and every question, answer and judge verdict is in research/runs/graph-v9.6-rival-agentmemory-k5. If the method is wrong, I would like to know.
What Graph-MIND is, briefly
A local memory shared by the coding agents in your terminal (Claude Code, Codex, Cursor, Gemini CLI, Copilot CLI, OpenCode). It stores every turn verbatim in SQLite on your machine, with no model call at write time, and retrieves with word search plus a small local embedding model. Open source (AGPL), one-line install.
The takeaway, whichever memory you use: ask for answer accuracy, on the same reader and judge.
