Retrieval-Augmented Self-Recall: The RAG Problem Nobody Talks About

Part 1 of Retrieval-Augmented Self-Recall — the research track behind Claude Code, Beyond the Prompt. All code is open source: RE-call.

Almost every RAG tutorial you've read solves the same problem: you have a pile of documents, a user asks a question, and you retrieve the chunks that answer it. Rank the right passage to the top, stuff it in the prompt, done.

There's a second kind of RAG that behaves completely differently, and almost nobody writes about it: an agent retrieving from its own memory.

I ran into it building the operational brain behind an automated trading system. That agent accumulates memory as it works — over 700 typed markdown memos, about 5 MB, re-indexed daily. Decisions, dead ends, calibration notes, "we tried this and it failed" postmortems. When the agent starts a task, it queries that memory: have we tested this before? what did we decide about X? is this still true?

The moment your knowledge base is the agent's own growing memory, the RAG problem inverts — and standard retrieval quietly does the wrong thing.

Why self-recall is not document QA

In document QA, there's a load-bearing assumption you probably never think about: the answer is in the corpus. Someone asked a question because the docs can answer it. Your whole job is ranking — surface the right chunk.

In self-recall, the most important queries are exactly the ones where the answer isn't there.

The agent asks "have we tried a mean-reversion filter on this market?" If the honest answer is no, never, a ranking-optimized retriever will still cheerfully return the three most cosine-similar memos — probably something about a different filter on a different market — and the agent, seeing confident results, concludes "yes, we looked at this." It just made a decision on a hallucination.

The failure isn't bad ranking. The top results might be the genuinely closest memos. The failure is that the system had no way to say "there's nothing relevant here."

Three failure modes unique to agent memory

Once you look at memory this way, three distinct failure modes show up that document-QA RAG never has to handle:

1. Hallucinating over gaps. The query has no real answer in memory, but retrieval returns the nearest neighbors anyway, and their presence reads as a "yes." The system needs to abstain — to flag "this is probably a gap" instead of pretending.

2. Re-litigating settled decisions. The agent proposes an idea it already tried and killed three months ago, because the "we decided against this, here's why" memo didn't surface at the moment of proposing. Memory that can't defend its own past decisions is doomed to relive them.

3. Acting on stale memory. A memo that was true in April is retrieved and treated as current in July. Without a freshness signal, old truth and current truth are indistinguishable — and in a system that touches money, that's expensive.

None of these are ranking problems. You cannot fix them by getting a better embedding model or a fancier reranker, because a better retriever just returns more confidently wrong results faster.

The reframe: abstention, not ranking

Here's the thesis of this whole series:

Document-QA RAG optimizes ranking. Agent-memory RAG has to optimize calibrated abstention — knowing when it doesn't know.

This matters because the metrics you've been trained to care about — MRR, nDCG, precision@k — don't measure abstention at all. They score how well you ordered the results assuming an answer exists. They are silent on the case that matters most in self-recall: the query with no answer, where the correct behavior is to return nothing and say so.

That's why reaching for an off-the-shelf RAG stack and pointing it at your agent's memory feels fine in a demo and rots in production. The stack is tuned for the wrong objective. It was never asked to abstain, so it doesn't.

RE-call: a reference implementation

To work through this properly I built and open-sourced RE-call — a retrieval engine designed around abstention from the start rather than bolted on after. The one-paragraph version, which the rest of this series unpacks:

• Storage & retrieval: PostgreSQL + pgvector as a single transactional store — dense vector search and sparse full-text search in the same database, fused with Reciprocal Rank Fusion, with an optional cross-encoder reranker.
• Three honesty guards: a gap_warning that fires when the best match is too weak to trust, a freshness signal that flags stale memory, and an anti-re-litigation check that surfaces closed decisions before the agent re-proposes them.
• Honest evaluation: a harness that measures not just ranking quality but a false-confident rate — how often the system fails to abstain when it should.

It ships as an MCP server, so an agent (Claude or otherwise) can query its own memory directly. That's the loop that closes back to my other series — this is the engine underneath "the memory file" and "semantic search."

What this series covers

Six parts, each a standalone piece of the problem:

1. This one — why self-recall is a different problem.
2. Hybrid RAG on nothing but Postgres — the architecture, and why I didn't reach for a dedicated vector database.
3. Teaching RAG to say "I don't know" — the three honesty guards, in detail.
4. Benchmarking retrieval and honesty — the eval harness, and why I measure a false-confident rate alongside MRR.
5. The gap threshold that didn't transfer — the finding that a single hard-coded abstention threshold is worthless across embedding models. This one surprised me.
6. The fine-tune that did nothing, and shipping it as an MCP server — an honest null result, then how the whole thing deploys.

A note on tone, because it's the point: this track reports what didn't work. A fine-tuning experiment that produced zero lift. An abstention threshold that fell apart the moment I changed embedders. In most domains those get buried. In this domain — calibration, honesty, knowing your limits — the negative results are the most useful thing I can hand you.

Next

Part 2 builds the retrieval core: dense plus sparse plus fusion plus reranking, all inside a single Postgres database, with pluggable embedders — and the argument for why, in 2026, you probably don't need a separate vector store to do this well.

Update (July 2026): the comments below did exactly what publishing is for. Two of them — the entailment-over-similarity argument and "supersession is a relation, not a property" — became measured experiments and shipped as RE-call v0.3: an opt-in entailment stage for the near-miss no threshold can catch, write-time supersession that beats even a steelmanned timestamp heuristic (83–100% stale-trust → 0.00), and a supersession lint. The full follow-up, with the commenters' names on it: What the Comments Taught Me (RE-call v0.3). The "three honesty guards" described above were v0.1's set — the current table has six.

Part 1 of Retrieval-Augmented Self-Recall. Code: RE-call (Postgres + pgvector, MIT). If you came from Claude Code, Beyond the Prompt, this is the engine under Part 1's memory and Part 5's search.