Every agent memory system I have read stores what was said and when. Almost none store where. The chunk knows its text, its embedding, a created_at, maybe a session id. It does not know that you learned it on a Wikipedia article, or typed it into a vendor onboarding form, or that it came out of a ChatGPT conversation you had on a specific page.
That missing axis costs you more than it looks like it should. Without it you cannot answer "what do I already know about the thing in front of me," you cannot forget one source without hunting chunk ids, and you cannot tell a user which of your stored facts came from their bank's website, which, the moment you have a browser extension, is the question that decides whether they trust you at all.
I spent two days adding that axis to Vodou's memory store. Here is what it actually took, including the four defects that every unit test in the repo said did not exist.
Three columns and a match endpoint that tiers by URL
Three columns on the memory chunk: source_url, source_host, source_ref. Then a match endpoint that, given a URL, returns what memory holds about it in four tiers.
// MCP-servers/Vodou-Console/src/api/page-match.ts — the tiers
// T1 exact page facts whose source_url matches this URL
// T2 this site facts from the same host
// T3 semantic facts that match the page's title, not its URL
// T4 documents Library docs saved from this page
On top of that: a browser extension that colors its own toolbar icon green when the current page has memory from it, a "Note about this page" box, insert-anywhere, typing suggestions, form fill from memory with a review card, and a per-site governance model that decides whether Vodou is allowed to look at the tab at all.
What happens when you switch tabs
The interesting design constraint is the last edge. The icon is green for T1 and T4 only. That was not the first thing I built.
Everything green, then four defects in build .74
Everything green, four defects
I finished the unit suites. page-match.test.ts, page-probe.test.ts, page-site-mode.test.ts: all passing, plus four new route tests for the note and link endpoints. Then I loaded build .74 as an unpacked extension with the DevTools Network tab open and used it like a person for twenty minutes: a real ChatGPT capture, a Wikipedia clip, a saved article, the keyboard shortcut.
Four defects. None of them visible to any test I had written.
A thirty-minute-old fact rendered as "18h old." The by-page query returned COALESCE(valid_at, created_at), and for a fact extracted out of a daily log, valid_at is the log day at local midnight. So a fact written at 12:30pm carried an instant of 00:00:00 and the age formatter did the arithmetic correctly on a value that was never meant to be read at that precision. The fix is not in the query, it is in the renderer: day-granular instants now render "today" / "yesterday" / "3d", never hours. A timestamp with a granularity is not a timestamp.
Two Wikipedia clips landed with an empty source_url. The right-click "Send selection to Vodou memory" handler sent the selected text and nothing else. The selection is the content, so the handler looked complete. It had no tab metadata, so both clips were stored with no page: findable by search, invisible forever from the page they came from. The whole feature, silently not working, on the exact path a user would try first.
The Library lane never stamped anything. There are two ways a page becomes a document: add_url and add_text. add_url wrote memory_sources.source_url and stamped its chunks. add_text did not. "Add this page to Vodou Library", the button an actual person clicks, routes through add_text. I had implemented the axis on the lane nobody uses. I mirrored it and hand-stamped doc #225 to match.
Then the fix broke the feature. Once that 111-chunk article was correctly stamped, it flooded "From this page." The panel filled with # Title\n\nSource: fragments, raw chunk headers, 111 of them, and buried the four facts I had actually clipped. Correct data, useless surface. Fact tiers now exclude doc:% entirely, and documents get their own list that shows each document once instead of each of its chunks.
Every one of these lived in the gap between "the function returns the right value" and "a person got what they came for." The tests asserted the first. Nothing asserted the second.
Four indicators in one evening, three rejected on sight
The icon was supposed to be the easy part. Show the user that this page has memory. I built four versions in one evening and threw away three.
A text badge with a count: Chrome's badge overlay is fixed-size and ate a 16px icon. Unreadable. A corner dot: too small to notice, and it does not say what it means. A pulse: motion in a browser toolbar reads as an unending alert. You notice it once and then it is a thing you are ignoring forever. The one that shipped is the icon itself drawn green, redrawn per tab on a transparent OffscreenCanvas from our own artwork.
That last one had its own trap. Hue-shifting the brand-blue pixels works. A flat source-atop fill does not: it paints over the white eyes in the logo and you get a green blob. MARK_MODE still keeps dot and pulse one line away, because I do not trust that I am done being wrong about this.
The harder question was not how to draw it but when. My first cut lit the icon for any tier hit. Two failures, both obvious the moment I used it:
- Include the site tier (T2), and the count swallows the icon on any site you have ever stored anything from.
- Include the title-only semantic tier (T3), and every Google results page lights up, because a results page's title contains your query, which is what you were searching your memory for.
Green means T1 and T4 only: facts and documents saved from this page. That is a claim narrow enough to always be true. An indicator that fires on a weak match is worse than no indicator, because a user calibrates on it once and then never looks again.
The permission model was the actual feature
The part I underestimated by the widest margin: this is a browser extension that reads the URL of every tab you open. That is not a feature detail. That is the whole trust question.
The gateway is the authority, not the extension. Site mode is off / suggest / collect. Banks, health portals, tax sites and sign-in hosts default to off: Vodou does not read the tab there until the user says otherwise. Resolution order is user rule, then sensitive default, then global default, and a rule covers subdomains.
Off is enforced in four places, not one: page-match returns empty without asking the daemon, page_probe returns no icon, and note and link both 403 unless the mode is collect. The extension-side typing suggestions ask the mode first and send nothing until it answers. Four tests on the resolver, four on enforcement.
Then mem forget --host, which is soft (sets invalid_at), covers subdomains, and counts Library documents rather than removing them. First panel click is a dry run that shows you the count; a second click within 8 seconds confirms. Verified live on w3schools.com: hide, 1, undo, 0.
The QA plan for all of this was two-phase and network-observable, which is the only kind of privacy test I now believe:
Phase A page memory OFF → switch tabs → assert ZERO page-match requests
Phase B page memory ON → accept disclosure → assert requests appear,
and the fact walnut-lantern-4417 shows under
"From this page" in the ChatGPT tab
Phase A is the important one. It does not test that the feature works. It tests that the feature is absent, from the network panel, where a skeptical user would look.
Test the privacy guarantee on the wire, not the return value
Your privacy guarantee is a claim about the wire, so test it on the wire.
Nearly every agent system I have seen tests its permission model by asserting the return value of a permission function. That is testing your intent. The user's actual question is "did you send my data anywhere," and the answer to that lives in the network panel, not in your unit suite. Write the test that asserts zero requests. It is a strange test to write, most test frameworks make asserting a non-event awkward, and it is the only one that answers the question that was asked.
Two corollaries, both of which cost me a defect each:
When you add a field, find every write path, not the one you were looking at. add_url and add_text both create documents. I stamped one. The one I missed was the one behind the button. Grep for every writer of the table, not every caller of the function you happened to open.
A stored timestamp has a granularity, and the granularity does not travel with it. A datetime column looks second-precise. A value derived from a log day is not. If you render it at a precision the data never had, you produce a confident lie: "18h ago" for something thirty minutes old. Carry the granularity, or render at the coarsest one you can guarantee.
Agent memory tiers say nothing about which page you were on
The public discussion of agent memory is almost entirely about tiers. The engineering playbook's agent-memory writeup splits long-term memory into episodic, semantic and procedural. Geodocs' pattern spec adds working memory on top and specifies consolidation, scoring, eviction and write-time PII redaction. Both are good. Both are about what kind of thing a memory is.
Neither has an axis for where the user was standing when it entered the store. That is not an oversight in the taxonomy so much as an artifact of the assumption underneath it: that memory arrives through your chat interface. Once memory can be captured from any page in a browser, provenance stops being metadata and becomes the primary key for two operations you cannot otherwise perform: "show me what you know about this" and "forget this source."
The Pockit production-memory guide is right that context windows are not memory and that persistence is the wall everyone hits. But durability cuts both ways, which is the point of "From Faulty Memories to Corrected Actions": persistence makes errors durable too, and deleting a bad memory leaves everything already derived from it in place. Their answer is dependency-guided rollback. My cheaper, dumber version of the same insight is that if every chunk carries its origin, "forget everything that came from this host" is one soft update instead of an archaeology project. Provenance is not just retrieval: it is the handle you need to undo.
Worth reading alongside these: agentmemory#648, a stability pass where two separate bugs turned out to share one root cause, a daemon wrapper surviving stop, and the dawnai memory-tooling commit, which adds a gated live-smoke suite for exactly the reason I am writing this post: some behaviors only appear against a real backend, so they gate a separate suite on a real key and keep it out of CI.
COALESCE lets a chunk's own page: token overwrite mem page-link
The stamping path has a known conflict. mem page-link writes the column directly, but re-indexing uses a COALESCE that prefers a page: token found in the chunk's own text. So a chunk whose text carries its own page token will revert to that token on the next sync, overriding the link. I know about it, it is in the plan, it is not fixed.
Form fill is two lanes and only one of them is deterministic. Learn-back, answers you accepted on this page before, or on this site, is exact and ordered, latest wins. Everything past that is one LLM call with a hard "only what memory supports, never invent" rule, which is a rule you write down and hope for. The review card exists because I do not trust that lane. Nothing is written to the page until you tick it, and it never submits the form.
Sensitive-host defaults are a list. Lists are wrong at the edges. A bank I have never heard of collects by default until somebody notices, and "somebody notices" is not a security model.
And the whole thing rests on one number I cannot measure from here: how many of those four live-only defects still exist in the paths I did not manually click for twenty minutes. Every unit suite in this repo is green. That was true before I opened the browser, too.
Source: Your Agent's Memory Has No Idea Where It Was by Chad Priest, from Building Vodou in Public.
