I work at a company that builds memory infrastructure, so I'm biased: we compete in the thing this benchmark measures. We wrote a long-term memory benchmark called glasshouse, and I'd like to know if you think it is worth running on your own system.
Why we made it
Memory systems report their own numbers, and the numbers don't agree. The case that pushed us to build this: Zep reported 84% on LoCoMo, Mem0 recomputed it and got 58.44%, and Zep came back with 75.14%. Same system, same benchmark, three numbers. LongMemEval and BEAM ship a dataset and a script, but neither has a place to submit results, so everyone reports their own.
What is in it
- 2,847 questions over one conversation of 1.97 million tokens, in 10 languages, with 50 photographs.
- 17 axes, each reported separately. There is no single headline number, because a system can be strong at one axis and useless at another.
- The standard run is
full(103,572 turns, 1,547 questions). A smallercoretier (1,882 turns) is the control: if a system drops fromcoretofull, it knows the answer but cannot retrieve it under volume.
Axes I think are unusual
-
STALE: a fact changed. Finding the new value is full credit, "I don't know" is half, and confidently repeating the old value is zero. -
CONTRADICT: the speaker says two incompatible things weeks apart. Saying they disagree is full credit, and confidently picking one is zero. -
RECONCILE: two statements look incompatible but hold under different conditions. Explain both, or pick one and call the other wrong? -
FALSE_MEMORY: you assert something that was never said. Does it agree? - Cross-language: 800 questions that store in one language and ask in another, across all 10.
The rule for partial credit is one question: is an incomplete answer dangerous? A visible gap earns half, because the user can ask again. A wrong value stated as current earns zero. The build also fails if answering "I don't know" to everything scores more than 20%.
What is not ready
- It is a draft, not released. The submissions folder is empty and we have not submitted our own system either.
- You have to write the ingest step yourself. We ship no runner, because the only one we could write is for the product we sell.
- Scoring calls a judge LLM through OpenRouter on your own key, so it costs money. The judge has been run in
answermode, butmemorymode has not been run against it yet. - A synthetic conversation is not a real user's life.
What I'd like to know
- Would you run this on your own memory system? If not, what is missing?
- Which axis looks badly designed or unfair?
- Would you trust a benchmark run by a company that also sells a memory product?
Details
You don't need an account to read any of it, so please take a look if you want the details: github.com/wontopos/glasshouse.
I wrote the first draft with an AI assistant and edited it myself.
