I Gave Claude Code a Brain. This Time I Let the Tests Judge It.

The first experiment measured whether Claude Code knew the right answer. This one measured whether the work actually succeeded.

In my previous experiment, I tested whether a memory layer could stop Claude Code from confidently repeating project-specific mistakes.

It did.

But there was an obvious limitation.

The tasks measured whether the answer was correct. They did not ask the agent to modify a repository, run the result, and pass a deterministic test.

So I ran the harder version.

This time, the agent had to do real work in a temporary repository. A checker decided whether the result was correct. No judge model. No subjective scoring. The task either passed or it did not.

Here is the result from the complete run:

RE-call improved success over the CLAUDE.md baseline by 22.2 percentage points.

The 95% confidence interval was +11.1 to +33.3 points.

The paired McNemar test gave a p-value of 0.000145.

This was not one lucky demonstration. It was 72 paired comparisons across 24 tasks and three seeds, with zero discarded cells.

What changed from the first experiment?

Each task asked Claude Code to create or modify files in a repository and run the result.

The benchmark included tasks involving things like:

• deterministic file ordering
• Unicode normalization
• migration naming
• shell quoting
• money rounding
• stable sorting
• line ending handling
• generated files and golden outputs

The important detail is that every task had a deterministic checker.

The agent could say "done" all it wanted. The checker only cared whether the repository ended up in the correct state.

I ran every task through three configurations:

1. Bare Claude Code
2. Claude Code with the project's CLAUDE.md
3. Claude Code with the same environment plus RE-call memory

The harness also verified that the memory tools were actually available before counting a session. If the memory layer was missing, the entire paired cell was discarded.

That happened zero times in the DeepSeek run.

RE-call helped where project memory mattered

The most useful comparison is RE-call versus the static CLAUDE.md file.

The static file produced a 36.1% success rate.

RE-call produced 58.3%.

That is a 22.2 point improvement.

The paired results were even more revealing:

• RE-call won 17 comparisons that CLAUDE.md lost
• CLAUDE.md won only 1 comparison that RE-call lost

The effect was strongest on the tasks classified as memory-sensitive. Across those eight tasks, RE-call improved success by 45.8 percentage points.

That is the pattern I was looking for.

Memory did not make every task easier. It helped when the task depended on a project decision, convention, or failure mode that was not fully present in the immediate prompt.

The surprising result was CLAUDE.md

The static CLAUDE.md configuration performed worse than the bare configuration:

The bare configuration won by 13.9 percentage points.

This is not what I expected.

It suggests that a static instruction file is not automatically a memory system. It can add useful guidance, but it can also add noise, constraints, or instructions that compete with the task.

RE-call is doing something different.

Instead of putting every historical fact into the permanent prompt, it gives the agent a way to retrieve relevant project history when the current task needs it.

That distinction matters.

Memory did not just improve the final score

The benchmark also measured what happened inside the memory mechanism.

In the DeepSeek run:

• The agent searched memory in 83.3% of eligible sessions.
• When it searched, it reached useful context 85.0% of the time.
• Overall, useful project context was reached in 70.8% of sessions.

So the result is not simply that "the model had more tokens."

The memory layer was being consulted, and in most cases where it was consulted, it reached something useful.

There is still room to improve retrieval. A search is not automatically a good search. The model needs to find the right historical decision, not merely retrieve something related to the words in the prompt.

The cost

The complete DeepSeek run cost an estimated $0.4964 at the captured API prices.

The three arms cost:

RE-call used about four times as many total tokens as the static prompt configuration.

That is the tradeoff.

The memory layer costs more on every task, including tasks where memory is unnecessary. In exchange, it can prevent the much more expensive failure mode where an agent confidently follows an outdated convention, repeats a known mistake, or spends several minutes debugging a problem the team already solved.

The next engineering question is not whether memory helps.

It is how to make the retrieval smaller, faster, and more selective.

What about GPT-5.3 Codex?

I also started the same model-freeze run with GPT-5.3 Codex.

That run is not a negative result for GPT.

The provider hit a credit and in-flight request limit while the benchmark was running. The harness correctly discarded the affected paired cells instead of treating missing sessions as failed model attempts.

Only 40 of the 72 paired cells were admissible, so I am not using that run to claim that GPT-5.3 Codex is better or worse.

The descriptive numbers pointed in the same direction:

But the run needs to be repeated with the provider capacity issue fixed before it becomes a valid model comparison.

That is an operational failure, not a model conclusion.

What this proves

This experiment does not show that memory makes an agent generally smarter.

It shows something narrower and more useful:

A production memory layer can improve the probability that an agent completes a real repository task correctly.

In this run, RE-call improved success by 22.2 percentage points over the static project file.

It also improved success over the bare configuration, although that gain was smaller at 8.3 points. That difference is important. The main value is not simply adding more instructions. The value is retrieving the right project-specific context when it matters.

The first experiment showed that memory can make answers more correct.

This experiment showed that the improvement survives contact with the repository.

That is the result I was actually waiting for.

What comes next

The next step is a clean competitor comparison:

• rerun GPT-5.3 Codex with sufficient provider capacity
• compare RE-call against other memory systems
• keep the same tasks, seeds, checkers, and admission gate
• publish discarded cells and operational failures alongside the scores

The goal is not to produce another flattering demo.

The goal is to find out whether a memory layer remains useful when the models, vendors, and retrieval systems change.

That is the benchmark I want to run next.