Tencent's Hy Memory Is Only as Good as the LLM That Writes It. I Measured It on LongMemEval

Tencent released Hy Memory 2.0 this week: an open framework (Apache-2.0), a 30B memory model and MemHub, an app that shares one memory across AI tools. I build something in the same space, so I wanted to know how it does on the two failures I care most about:

1. Answering with an outdated version of a fact that has since changed.
2. Forgetting who said what, especially what the assistant itself said.

Disclosure: I'm the author of Aura, an open-source memory library with a Windows app. I wrote down the setup and pass lines before running, and I report the results that went against me too.

How I tested

I used LongMemEval, a public benchmark. Each question comes with a user's chat history of about 50 conversations, and the answer is hidden somewhere in it. I took three of its six question types, 20 questions each:

Both systems got the same history and the same question. For both, the same model read the retrieved memory and answered (Gemini 3.8 Flash), and the same model graded the answer with LongMemEval's official judge prompts.

• Hy Memory 2.0.1, its default pro mode. Hy uses an LLM to turn each conversation into facts and to reconcile them with older ones. Tencent's own 30B model isn't served by any API I could use; the framework works with any OpenAI-compatible model (its README uses DeepSeek). So I ran it twice: with Gemini 3.8 Flash, and with Claude Opus 5.5. Opus is a fair stand-in: Tencent's model card compares its model with Claude Opus 5 and gets nearly the same average (55.5 vs 54.6).
• Aura 1.62.0 keeps every turn as it was said, with the date and who said it, and finds the 10 most relevant. It uses no LLM when writing; embeddings run locally.

Result 1: with a cheap writer, Hy answered 43%

32 of Hy's 34 misses were "I do not have that information".

The first objection I expected: Hy's 10 facts are shorter than Aura's 10 turns (about 2,000 vs 5,000 characters), so maybe Hy just had less to read. I searched Hy's stores again with a bigger limit, without writing anything again. With as much text as Aura it scored 45%. With 40 memories (about 7,900 characters) it still scored 45%. The information was lost when the conversations were turned into facts, not at search time.

Result 2: with Claude Opus writing, Hy caught up on updated facts

Opus as the writer cost about $5 per question, so I ran 10 questions per type, not 20.

Knowledge update, the same 10 questions:

This one went against me. With a strong writer, Hy tracks changed facts as well as Aura does. It keeps the old value as history ("Current: 25:50 … History: 27:12"), which is the right design. Both missed the same question: two different counts on the same day.

Assistant said, 8 questions (my Anthropic credit ran out during two of the ten, so those two are excluded):

Opus helps a lot here: it does store some of the assistant's advice as memories ("The assistant suggested…"), which Flash never did. Still, half of what the assistant said never made it into memory. All four misses were "I do not have that information", even with 40 memories retrieved.

What the same accuracy costs

For one user history of about 50 conversations:

That's the real trade-off. Hy's quality is bought at write time, with an LLM reading every conversation. Use a cheap model and it drops to 55–60% on changed facts. Use Opus and it matches a memory that never calls a model, at about $5 per user history.

Why: rewriting conversations into facts loses things

Hy compresses each conversation into statements about the user ("The user is interested in learning about and cooking classic Peruvian cuisine"). That makes memory small and tidy. But it throws away whatever the writer doesn't consider a fact about the user: what the assistant recommended, details that seemed minor, sometimes the newer version of a fact. Examples from the run with Flash:

• "Did I finish The Nightingale?" Hy said the book was set aside as too emotional; later the user had finished it.
• A previous goal of 100 was overwritten by the new goal of 150, so "what was my previous goal?" had no answer.
• "Which beer did you recommend for the recipe?" The recommendation was the assistant's, so it wasn't stored.

Aura goes the other way: it keeps what was said, with who said it and when, and leaves deciding what matters to the model that answers. That costs more storage and gives the reader more text. But nothing has to be guessed at write time.

Limits

• Three of six LongMemEval types, chosen because they match the two failures above. I didn't test multi-session reasoning, temporal reasoning or preferences, where summarising into facts may well help.
• Small samples. 60 questions in the main run; 10 and 8 with Opus. Each question is 5–12 percentage points.
• Not Tencent's own model. Their 30B model might write better than Flash. Opus is the closest comparison their own model card offers.
• Pro mode only. I didn't test ultra (graph mode).
• One reader and judge model (Gemini 3.8 Flash), the same for both systems.
• Hy failed to store 1.3% of conversations in the main run (it sent empty text the model rejected), worth at most 1–2 questions.

Takeaway

If you evaluate a memory system that summarises with an LLM, check two things on your own data: which model it needs to be good, and what that model costs per user. And ask whether it keeps what the assistant said, because "what did you recommend last time?" is a question users do ask.

Aura is open source (MIT): github.com/teolex2020/aura-memory. The Windows app shares one local memory across Claude Code, Cursor, Codex, Gemini CLI and others: aurasdk.dev/desktop.