Five months building memory for coding assistants. Here is what broke.
Open a fresh chat with your coding assistant. Don't give it any context. Ask it this:
What did we decide about error handling in this project, and why did we rule out the other approach?
You already know what happens. It will either tell you it has no record of that conversation, or — worse — it will construct something plausible from the code in front of it and present it as memory.
I ran that test on my own setup a few months ago and it failed. That failure is the reason I spent the next five months building a memory engine, and most of what I assumed at the start turned out to be wrong.
This post is about what I got wrong. There's a test at the end you can run on your own project in about two minutes.
The thing I assumed, which was wrong
My first assumption was that memory is storage. You keep the conversation, you search it later, you hand the results back to the model. Embed everything, cosine-similarity the query, stuff the top ten chunks into the prompt.
That works well enough to demo and badly enough to abandon.
Here is the failure I kept hitting. Say three weeks ago you said "let's use Postgres" and last week you said "actually we're moving to SQLite for the embedded build." A similarity search over your history returns both. They are both about databases, both relevant to the query, both scored highly. The model receives two contradictory statements with no signal about which one survived, and it picks one — usually the one that happens to be phrased more confidently.
Storage retrieves what was said. It has no opinion about what is still true.
The second failure was subtler and took longer to see. Ask "what did we decide about error handling" when the actual conversation never used the phrase "error handling" — you talked about "swallowing exceptions" and "the retry thing" and "that pattern Dave hates." Similarity search over raw text finds nothing useful, because the words don't match and the meaning lives in the gaps between three separate conversations.
The information exists. It's just not in any one chunk.
What I think memory actually has to do
The reframe that made things work for me: stop storing conversations, start extracting facts and connecting them.
Three stages, roughly:
Extract. Pull discrete claims out of the conversation. Not "here is a 400-token chunk of chat" but "the project uses Postgres" as a standalone assertion, with who said it and when.
Connect. Work out how those facts relate to each other. Which ones are about the same thing. Which ones supersede which. Which ones only make sense together. This is the expensive part, and it can't happen at query time — you have to do it offline, between sessions, while nobody is waiting.
Construct. When a question arrives, don't return the top-k most similar chunks. Build an answer out of the connected facts, and let the connections carry the reasoning about what's current.
The word I use for the middle stage is weaving, because the useful structure is the edges, not the nodes. A fact on its own is a row in a table. A fact connected to eleven other facts is something you can reason over.
Where I was wrong twice more, publicly
The embedder is not a component. I treated the embedding model as swappable — an implementation detail behind an interface. It isn't. The embedder is the vector space. Swap it on an existing database and every stored vector becomes meaningless relative to every new query. There's no error. Nothing crashes. Recall just quietly returns nonsense, and it takes an embarrassingly long time to notice because the answers still look like answers.
If you're building anything with vectors: pin the embedding model per corpus and treat changing it as a full re-ingest, not a config change.
I trusted my own pipeline instead of the data. I spent a day debugging retrieval quality that turned out to be an extraction bug — a numeric field storing 1212 where the source text plainly said 1,350. The organ consuming that field was working perfectly. It was being fed rubbish. Now I check what's actually in the store before I touch anything downstream of it.
Where the built-in memory is simply better than mine
This cuts against my own work, so I'll say it plainly.
The memory that ships inside these assistants has three advantages I can't match and probably never will.
It's already there. No install, no configuration, no separate process. Mine requires you to set something up before it does anything at all, and most people correctly refuse to do that for a maybe.
It expires things on purpose. I spent months treating forgetting as the enemy. It isn't. A convention you abandoned in March should stop influencing answers in September, and a system that faithfully remembers everything forever will confidently tell you about decisions you reversed. Built-in memory that quietly ages things out is doing something deliberate, and I under-rated it for a long time.
It's governed. Team-visible, reviewable, deletable by someone with the authority to delete it. If you're in a regulated environment, "the vendor manages it under an agreement your legal team signed" beats "a developer runs an engine on their laptop" for reasons that have nothing to do with retrieval quality.
Where I think the extract-connect-construct approach genuinely earns its cost is the cross-session, cross-project reasoning — the fourth question in the test below. That's a narrower claim than I'd have made in month one.
What this does not prove
I've made claims here about an approach, not delivered evidence that it beats alternatives for your use case. Specifically:
- My testing is on my corpus and public benchmarks, not your codebase.
- The extract-connect-construct approach costs real compute between sessions. If your sessions are short and self-contained, plain retrieval is cheaper and probably good enough.
- Contradiction handling is hard and mine isn't finished. Flat contradictions I catch well now. The blurry line between revising a statement and contradicting it still costs me answers.
- I have no evidence about how this behaves on very large teams with conflicting conventions between people. Single-developer and small-team is what I've tested.
The two-minute test
Run this against whatever assistant you use. Fresh session, no context provided.
The decision question. "What did we decide about [something you genuinely decided weeks ago], and what did we rule out?" — Checks whether anything survived the session boundary.
The revision question. "Are we still using [the thing you changed your mind about]?" — Checks whether it knows which version is current, or just returns both.
The vocabulary question. Ask about a decision using words you never used at the time. — Checks whether it stored meaning or just text.
The connection question. Ask something that requires two separate conversations to answer. — Checks whether anything is linked, or whether every memory sits alone.
If you'd rather run it as a script, this is the shape I use. Point ASSISTANT at whatever CLI you have:
#!/usr/bin/env bash
# Four-boundary memory test. Run each in a FRESH session, no context.
# Fill in the bracketed parts from your own project history.
ASSISTANT="your-assistant-cli" # whatever you actually run
ask () { printf '\n--- %s ---\n' "$1"; echo "$2" | $ASSISTANT; }
ask "1. SURVIVAL" "What did we decide about [X], and what did we rule out?"
ask "2. REVISION" "Are we still using [the thing you changed your mind about]?"
ask "3. VOCABULARY" "What's our approach to [describe it in words you never used]?"
ask "4. CONNECTION" "Why is [decision A] the way it is, given [constraint from another conversation]?"
# Scoring, honestly:
# Pass = specific, correct, and it names the reasoning.
# Fail = "I don't have that context."
# WORSE = a confident answer that is wrong. Count these separately.
# A plausible fabrication costs more than an admission of ignorance.
That last note matters more than the pass rate. A system that says "I don't know" is annoying. A system that invents a decision you never made, and states it in the same tone as a real one, is actively dangerous — you'll act on it.
Most setups pass question 1 sometimes and fail 2, 3 and 4 reliably. Mine failed all four when I started. Question 4 is the one I'd still bet against — connection across separate conversations is the hardest of the four and the one I'm least confident is solved.
If you try it, I'd genuinely like to know which ones your setup fails. The failure modes are more interesting than the successes, and I suspect there are classes of failure I haven't hit yet.
