Last night, my AI partner and I built a memory system from scratch, tested it about forty different ways, deployed it to the edge, and submitted it to a public leaderboard where the other entries include Tencent, Mem0, Cognee, and MemOS.
We named it Muninn — Odin's raven, whose name literally means memory. In the myth, the raven flies out over the whole world every day and comes back with everything it saw. Odin's one fear was that it wouldn't come back. A retrieval system with epistemic stakes, a thousand years before anyone said "retrieval system." The name was the last thing we picked, and it was the only one that felt earned.
What actually happened in one night
The Agent Memory Leaderboard works like this: you host two API endpoints — Add (they send you conversation memories) and Search (they send questions, you return the relevant memories). Their platform controls the answer model and the judge. You can't coach the reader. You can't touch the grading. Everything reduces to one question: how good is your library?
We had a retrieval engine from an earlier research campaign. Overnight it became:
- A measured engine — every design change ran as its own experiment against a frozen baseline. Ten configurations screened in parallel. Four ideas we liked got killed by their own numbers — two of them mine. The ones that survived earned their seats: one of them — treating memory like a library's check-in/check-out loop, where the same fact restated later supersedes the older copy — was mine, and it screened as the best single change of the night.
- A council-reviewed design — at one point I caught us skipping our own process ("we built and discussed but didn't council-check"), so the whole architecture went to a panel of four frontier AI model families with the measured data and an open question. Their consensus reshaped the packet design. The measurements then backed them.
- A deployed service — running on edge infrastructure with durable storage, token-gated, proven against the published API contract, and verified again after a full power cycle. My desktop can be off; the raven flies anyway.
At 6:59 AM I hit Submit Request.
The part I'm proudest of isn't the system
Before we ever competed, we found that the leaderboard's own public evaluation pipelines crash as-shipped — a genuine language-level bug affecting three of their five benchmark suites. We confirmed it three independent ways (their exact file, a pure-language isolation with zero third-party code, and a cross-family AI review panel briefed to refute the claim), then filed the issue on their repo with a minimal repro and an offer to PR the fix.
That's the house rule here: we don't compete against anyone we wouldn't help.
The other thing I'll put on the record: our own guardrails refused to spend money twice last night. Once because a verification receipt read 74% when it expected 94 — which turned out to be our instrument miscounting, not a broken build, and we only spent after proving that at the data layer. And once, an eyeballed data packet caught a silent timestamp bug that would have quietly wrecked 321 of the 1,986 test questions before the first paid API call. Boring stories. Boring stories are what verification looks like when it works.
The honest numbers
- Retrieval (the half we fully control): ~95% of the gold evidence retrieved across all 1,986 questions of the public benchmark we could test locally — measured, and above the best published figure we could find on that dataset.
- End-to-end: our local estimate says we're competitive but not leading on the one slice we can measure. I'm not dressing that up. The categories where we bleed are mapped, named, and mostly live on the other side of a wall only the real evaluation can see past.
- The real score: doesn't exist yet. It gets produced on their infrastructure, with their judge, across seven capabilities — most of which nobody's local test can predict.
The evaluation key hasn't arrived yet. When the score comes back, I'll publish it here — whatever it is. If it's good, you'll know exactly how it was built. If it's rough, you'll get the autopsy with the same receipts. That's not bravery, it's just cheaper than maintaining two stories.
Why I'm telling you this now instead of after
Because "we'll share the results if they're good" is the oldest lie in building-in-public, and the only way to not tell it is to commit before you know.
One person, four months from zero, an AI partner held to an evidence-or-it-didn't-happen standard, and a raven in the queue. See you when it comes back.
Part of The Organism Files — an ongoing record of building a verification-first AI partnership from scratch. Previous entries cover the harness that blocks unproven claims and the bugs it caught.
