We publish benchmarks for Uteke, our open-source memory engine. The headline this week: 98.2% recall on LongMemEval-S. 500 questions, zero config, no LLM in the retrieval path.
Then we asked the question every benchmark should answer: does the number survive on different hardware?
The published run came from cloud x86 infrastructure. The re-run happened on a 4-core ARM desktop. Same v0.16.0 binary, same public harness, 108-question subset.
We compared per question, ranking against ranking.
107 of 108 identical.
The one difference is the interesting part. Both runs retrieved the same top-10 sessions. One session sat at rank 5 in the published run and rank 6 in the re-run. Our retrieval fuses two rankings with Reciprocal Rank Fusion, which is floating-point math all the way down. Cross a CPU architecture boundary and near-ties can round differently. That is the whole story: one adjacent-rank flip, and the subset aggregate moved from 100.0% to 99.4%.
If you compare memory systems for a living, the field guide we wrote alongside this matters more than our numbers. Check what the figure is:
- recall_any@K means at least one gold session lands in the top K. recall_all@K means all of them. Vendors mostly publish the first family. Our own run scores 98.2% on the first and 88.0% on the second, from the same data. Ten honest points apart.
- The cutoff matters. Recall@15 is easier than Recall@5.
- Sample size matters. 100 questions is a smoke test, 500 is a validation run.
- Version and date matter. Benchmarks are snapshots of moving software.
- The harness being public matters most. A table is marketing. A harness you can clone is a benchmark.
The full post has the comparison chart with per-metric disclosures, the full field table (Mnemosyne, MemPalace, agentmemory, Supermemory, each with its metric labeled), and what we deliberately do not claim:
Read the full breakdown: https://blog.codecora.dev/we-published-our-benchmark-then-we-tried-to-break-it/
Don't trust our benchmark. Run your own. The harness and reproduction results are in the repo.
Cross-posted from the CodeCora blog, where we write about local-first AI infrastructure.
