Every agent framework ships with the same pitch: give it memory, watch it get smarter. Store the conversation history. Store the tool outputs. Store every mistake and every fix. Feed it all back in next time and the agent will learn from its past.
Turns out that's not how it works.
A team at IBM Research ran the experiment properly -- testing agentic memory across eight different models -- and the result should make you rethink whatever memory setup you're running right now. More stored experience did not reliably make agents better. In some cases it made them worse. According to Gigazine's summary of the work, selecting only the necessary information, rather than dumping the full history back into context, produced better performance -- and which approach won depended heavily on the model.
That's the opposite of the story most of the industry tells.
The assumption everyone's been running on
Walk through any agent memory pitch deck and you'll hear a version of: "distill lessons from past runs, store them, retrieve them later, agent performance compounds over time." It sounds obvious. It sounds like how humans learn. It's also the exact premise we leaned on in Breaking the Memory Wall when we looked at giving open-source agents Claude-level recall.
The IBM team's framing cuts right through that assumption. Equipping an agent with memory sounds simple -- distill lessons, store them, retrieve them -- but the actual behavior of the agent once you do that is not simple at all. Some models degrade when you hand them a big pile of prior context. Others need it. There's no universal answer, and that's the uncomfortable part for anyone who's been treating "add a memory layer" as a solved problem you bolt on and forget.
Memory isn't free -- it's a cost you're paying every turn
Here's the part that should sound familiar if you've read our piece on the VRAM currency problem. Hardware memory and agent memory have the same failure mode: you can't just add more and expect linear returns.
On the hardware side, a bigger context window costs you real VRAM, real bandwidth, real latency per token. We saw the same tradeoff pattern when we compared DDR5 6400 against DDR5 8000 on a Ryzen 9 build -- more bandwidth helps, but only up to the point where something else in the pipeline becomes the bottleneck. Past that point you're just burning cycles moving data the model doesn't actually use.
Agentic memory behaves the same way, except the bottleneck isn't silicon -- it's attention. Every token of stored memory you inject into the prompt is a token competing for the model's attention budget. Stuff the context with everything the agent has ever seen and you're not giving it more knowledge. You're diluting the signal. The model has to figure out which of the fifty prior tool calls actually matters for the task in front of it right now, and that's a genuinely hard retrieval problem disguised as a "just give it more context" solution.
This is exactly what the IBM researchers found when they varied how much memory they fed into each of the eight models. More wasn't better by default. It was better only when the memory was curated -- when the agent got the right slice of its past, not the whole archive.
So what actually decides the answer?
Three things, based on how the experiment framed the problem:
The model itself. Different models handle long, noisy context differently. Some are more robust to irrelevant information sitting in the prompt; others get pulled off track by it. If you're benchmarking memory strategies on one model and shipping the result as a universal best practice, you're extrapolating from a sample size of one.
The task shape. A coding agent debugging the same repo over multiple sessions benefits from precise, narrow memory -- the specific bug it hit last time, the fix that worked, the file paths involved. A research agent synthesizing across many unrelated sources benefits from broader recall. Same memory system, different retrieval strategy, different outcome.
How the memory gets selected, not just how much exists. This is the real headline buried in the IBM result. The question isn't "how big should the memory store be." It's "how good is your retrieval." A huge memory store with sloppy retrieval loses to a small memory store with sharp retrieval, and the gap isn't small.
That last point is where the open-source memory tooling landscape actually splits into two camps.
Two philosophies, two frameworks
If you go looking for a memory layer to bolt onto your agent today, you'll run into Mem0 and Cognee almost immediately -- they're the two most visible open-source options, and they solve the "what to remember" problem in genuinely different ways.
Mem0 is built around a simple add/search API. You push memories in, you pull relevant ones out, personalization is fast to wire up. It's the pragmatic choice when you want memory working this afternoon.
Cognee goes further. It's built to extract structured knowledge from messy, heterogeneous sources -- documents, images, audio, Slack threads -- and turn that into a queryable knowledge graph. That's a heavier lift, but it buys you something Mem0's flat retrieval doesn't: relationships between facts, not just facts.
Neither one is "the memory solution." They're two different bets on what your retrieval problem looks like. And per the IBM result, retrieval quality is the actual lever that determines whether memory helps or hurts. Picking Mem0 when your agent needs relational context between entities, or picking Cognee when you just needed a fast key-value recall layer, will cost you more than picking neither.
We covered a version of this same tension in The Claude Code Memory Gap, where the problem wasn't the absence of memory -- it was persistent storage that didn't map cleanly onto how the agent actually needed to retrieve it later. Storage and retrieval are not the same problem, and treating them as one is how you end up with a memory system that technically works and practically doesn't help.
What this means if you're building an agent right now
Stop treating memory size as the metric. It's the wrong axis. If your agent's context window has room for 200K tokens of history, you do not want to fill it with 200K tokens of history by default. You want a retrieval step that decides, per task, what actually matters -- and you want to test that retrieval step against your specific model, because the IBM findings make clear that behavior isn't portable across models.
Concretely:
- Benchmark memory strategies per model, not once. A memory setup tuned on one model's behavior may actively hurt a different model in the same pipeline.
- Treat retrieval as the product, not the store. Whether you're using Mem0's add/search pattern or Cognee's knowledge graph approach, the quality of what gets selected matters more than how much is available to select from.
- Watch for the failure mode where memory looks like it's helping but is actually adding noise. An agent that gets slower or less accurate as its memory store grows is telling you something -- usually that everything is getting shoved into context regardless of relevance.
- Match memory architecture to task shape. Narrow, repetitive tasks (the same codebase, the same support queue) want tight, specific recall. Broad synthesis tasks want structured, relational memory.
This also connects to the tooling layer underneath all of it. If you're building agents on something like the Anthropic Agent SDK, the memory decisions you make sit on top of whatever context-management primitives the SDK gives you -- and those primitives are only as good as the retrieval logic you put in front of them. The SDK hands you the plumbing. It doesn't tell you which eight models will behave differently when you fill that plumbing with a year of stored agent history.
The actual answer to the headline question
How much memory does your agent need? Less than you think, selected better than you're currently selecting it.
That's an unsatisfying answer if you wanted a number -- 10 turns, 50 turns, 4K tokens of summary. The IBM Research work is useful precisely because it refuses to give you that number. It shows performance moving in different directions depending on the model, which means anyone selling you a fixed memory budget as a best practice is selling you something that was benchmarked on a different setup than yours.
The practical move is to treat memory the way you'd treat any other resource with a cost curve -- VRAM, bandwidth, tokens per second. More isn't a strategy. Measured, curated, retrieval-first memory is. Build that, test it against your actual model, and you'll end up with an agent that remembers what matters instead of one that remembers everything and gets worse for the privilege.
