An agent can fail because it missed the right material. It can also fail because I gave it the wrong material with confidence. The worse case is quieter: the wrong material sits beside a successful task, then looks useful afterward.
A skilled user may ask for architecture notes because they know where to look. Those notes can travel with a good result even when they did not cause it. If the system learns from that trace alone, the next session may get extra text that feels justified and still wastes the agent's attention.
That failure has a name and a direction. The same expertise that makes someone request the right document also makes them likelier to finish the task without it. Skill causes both the request and the outcome, so skill is a common cause sitting upstream of the thing I am trying to measure. A system that reads the trace and credits the document has attributed to the context what belonged to the person. This is not noise that averages out as traffic grows. More sessions from confident users make the estimate tighter and no less wrong, which is the property that makes it dangerous: the system becomes more certain of a relationship it never established.
There is no way to subtract that bias afterward from the trace alone, because the trace does not record why the material was requested. The only cheap instrument that removes it is deciding who gets the material by coin flip instead of by request. Randomization makes assignment independent of skill, so whatever difference survives between the two branches is attributable to the material rather than to the person holding it.
This is the problem I built the active experimentation layer for in Zero Context Loss (ZCL). ZCL is the context learning platform I use for my AI agents. This post is about one part of it: the evaluation layer that tests changes to context provisioning while the agent is doing real work. It is also, by the end, an honest account of how much that layer can currently prove, which is less than its own vocabulary suggests.
1. A context change becomes a hypothesis
The active learning code lives in zcl_core/learning/active_learning.py. The central object is Hypothesis. I made the candidate change carry its own treatment, control, task type, metric, and minimum sample threshold.
That shape is deliberate. Reordering two documents has a different target from including a document whose value is uncertain. The first can be judged by time. The second can be judged by success. If those choices stay implicit, the system can promote a vague improvement without saying what improved.
Declaring the metric before the data arrives matters more than it looks. A hypothesis that names its outcome in advance can only be judged on that outcome. A hypothesis that stays vague can be judged on whichever of success, duration or token count happens to have moved, and something almost always has. Writing the metric into the object is the cheapest available guard against choosing the comparison after seeing the result.
The hypothesis object uses a UUID, a type string, a description, the task kind, treatment and control payloads, a metric name, and min_samples set to 20. The default threshold sits on the object because I wanted the brake to travel with the proposed change. Section 6 returns to whether that brake is connected to anything.
The generator in ActiveLearning stays close to context the system can actually apply. It looks at documents already used for a task type, then builds order hypotheses from the top five documents against the next slice. It also asks for uncertain documents and creates inclusion tests for up to three of them. Each hypothesis takes its id from uuid4() — called, not referenced.
A context system with limited traffic can burn every session it has on combinations that will never gather enough evidence to matter, so the generator stops early and some pairings never become live candidates at all, and I gave that coverage up on purpose. The generator is not there to enumerate everything that could be tested. It is there to keep the number of live tests small enough that each one can actually finish.
2. Assignment happens while context is being built
The integration point is zcl_core/provision/context_provider.py. The context provider creates the session id with uuid4(), decides whether the request should explore, asks value attribution for learned document values, handles cold start, then filters the selected document ids through causal evidence before the final bundle goes back to the agent.
Assignment has to happen before the agent consumes the material, because the branch has to be decided by the coin rather than by the request, and that ordering is doing more work here than anything else in the file. A report run later can describe what happened, but by then the material has already been chosen by whatever mixture of habit, skill and retrieval score produced it, and the confound from the opening is already baked into the data. Deciding during provisioning is what converts an observation into an experiment.
The returned ContextBundle carries is_experimental, experiment_id, and experiment_group. Those fields are plain bookkeeping, and they are the difference between analysis and guesswork. If a session was in treatment, the bundle says so. If it was in control, the evaluator does not have to reconstruct that from log order.
flowchart TD
candidate[Candidate context strategy] --> assignment[Randomized assignment]
assignment --> treatment[Treatment context]
assignment --> control[Control context]
treatment --> outcome[Task outcome]
control --> outcome
outcome --> evaluation[Statistical evaluation]
evaluation --> promoted[Promoted strategy]
evaluation --> retired[Retired strategy]```
The diagram lies in one place, and it is the place most likely to matter. Randomizing sessions treats sessions as independent units. They are not. The same person returns, so their skill appears in both arms across the week rather than being held constant within one. Learned document values also update between sessions, which means the control arm is not fixed while the experiment runs; the baseline drifts under the test. Randomizing the person rather than the session would remove the first problem, and freezing learned values for the duration of an experiment would remove the second. The current design does neither, and the effect is that the estimate is noisier than a clean trial of the same size.
## 3. How much of the traffic gets spent on trials
`ActiveLearning` starts with `base_exploration_rate = 0.2`, uses `uncertainty_multiplier = 0.5`, and caps the final probability at 0.4. These are configuration values in the implementation. They are not presented as observed production rates, throughput measurements, or hardware-dependent results.
The decision is small: compute task uncertainty for the organization, add the uncertainty bonus to the base rate, cap it, then compare that probability with `random.random()`.
The uncertainty calculation is scoped by organization. That stops one organization's history from making another organization's context appear more certain than it is. In the helper, fewer than 10 sessions returns maximum uncertainty. After that, the code reads outcomes for the task, computes success-rate variance as a Bernoulli variance, and reduces the result as sample count grows using the log of the count. With no outcomes, it uses 0.25 as maximum variance.
It is a cheap signal for where to spend trials, and it makes no claim at all about why a document helped.
There are two traditions tangled together in that decision, and they do not want the same thing. Spending more trials where uncertainty is high is bandit reasoning, and a bandit's goal is to minimise regret: give as many sessions as possible the best-known context while still learning. A controlled experiment has a different goal, which is an unbiased estimate of an effect, and it is happiest with a fixed allocation decided in advance. The two are not the same discipline. Adaptive allocation is known to bias the naive difference in means, because the amount of data each arm receives depends on how the arm has been performing.
Here the two are only loosely coupled: the uncertainty rate governs whether a session explores at all, while assignment within a live experiment is a fair split. That keeps the bias small. But the honest description of this layer is that it uses a bandit to decide when to run trials and a fixed randomization to run them, and if the exploration rate ever starts responding to the results of a specific live experiment, the estimator stops being trustworthy.
Capping it at 0.4 means the areas with the least evidence still spend most of their sessions on the path already known, which slows learning down badly. I kept the cap anyway. Without it a sparse task type turns into churn, every request treated as a trial, and nothing ever settles into a default.
## 4. The schema keeps the branches separate
The experiment tables are defined in `migrations/004_experiments.sql`. The migration separates the proposed change from the per-session assignment. One table stores the hypothesis text, task type, treatment, control, status, results, and timestamps. The other stores the session, experiment id, group name, and assignment time. Both take their ids and timestamps from `uuid_generate_v4()` and `NOW()`, called rather than named.
```sql
CREATE TABLE zcl_experiments (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
hypothesis TEXT NOT NULL,
task_type VARCHAR(100) NOT NULL,
treatment JSONB NOT NULL,
control JSONB NOT NULL,
status VARCHAR(20) NOT NULL DEFAULT 'active', -- active, completed, cancelled
results JSONB,
started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
completed_at TIMESTAMPTZ,
CONSTRAINT valid_status CHECK (status IN ('active', 'completed', 'cancelled'))
);
CREATE TABLE zcl_experiment_assignments (
id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
session_id UUID NOT NULL REFERENCES zcl_sessions(id) ON DELETE CASCADE,
experiment_id UUID NOT NULL REFERENCES zcl_experiments(id) ON DELETE CASCADE,
group_name VARCHAR(20) NOT NULL, -- treatment, control
assigned_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE(session_id, experiment_id),
CONSTRAINT valid_group CHECK (group_name IN ('treatment', 'control'))
);
JSONB is PostgreSQL's binary JSON type. I used it here because treatment and control payloads vary by hypothesis type. Ordering two documents is shaped differently from excluding one document, and forcing those into a rigid column set would add ceremony without improving the evaluator.
The two constraints are doing real work, and they are doing the kind of work that is easy to skip and expensive to skip. UNIQUE(session_id, experiment_id) means a session is assigned once to a given experiment. Without it, a retry or a duplicated provisioning call would let one session contribute two rows to the same arm, which inflates the sample count with correlated data and makes the test more confident than the evidence warrants. The group check keeps the result set to treatment or control, so a typo cannot open a third bucket that then silently reduces the size of both real arms.
The migration also adds indexes for task type, status, active experiments, assignment lookup, session lookup, and group lookup. The write path needs to find active tests during provisioning. The read path needs to count assignments when judging results. Those are different access patterns, so the schema names both.
5. Outcomes are measured after the task
Retrieval can produce tidy scores before the agent acts. Rank, similarity, token fit, and document presence all arrive early. None of them says whether the work succeeded.
This is the gap the title is about, and it is worth stating plainly rather than implying it. The standard instruments for retrieval quality — recall at k, mean reciprocal rank, normalised discounted cumulative gain — all score a ranking against a set of documents somebody labelled relevant in advance. They answer whether retrieval found what a human said was relevant. Agent memory has to answer something else: whether the material changed what the agent did. Those questions come apart precisely where the interesting failures live. A document can be topically relevant, rank first, be judged relevant by any labeller, and still cost the agent attention it needed elsewhere. Recall at k cannot see that, because the harm is not in the ranking. It is in what happened next.
Asking a model to grade the retrieval has the same shape. It scores the plausibility of the material against the request, which is a judgement made before the work and without knowing how the work went. Both instruments measure the retrieval, and both stop exactly where the question starts.
The migration defines get_experiment_outcomes. It joins experiment assignments to recorded outcomes, grouped by branch. The function returns group name, total sessions, successful sessions, success rate, and average time in minutes.
CREATE OR REPLACE FUNCTION get_experiment_outcomes(p_experiment_id UUID)
RETURNS TABLE (
group_name VARCHAR,
total_sessions BIGINT,
successful_sessions BIGINT,
success_rate FLOAT,
avg_time_minutes FLOAT
) AS $$
BEGIN
RETURN QUERY
SELECT
ea.group_name::VARCHAR,
COUNT(*)::BIGINT as total_sessions,
SUM(CASE WHEN o.success THEN 1 ELSE 0 END)::BIGINT as successful_sessions,
AVG(CASE WHEN o.success THEN 1.0 ELSE 0.0 END)::FLOAT as success_rate,
AVG(o.time_minutes)::FLOAT as avg_time_minutes
FROM zcl_experiment_assignments ea
JOIN zcl_outcomes o ON ea.session_id = o.session_id
WHERE ea.experiment_id = p_experiment_id
GROUP BY ea.group_name;
END;
$$ LANGUAGE plpgsql;
This is where the layer stops being retrieval self-scoring. An inclusion hypothesis can be judged on whether the session succeeded; an order hypothesis is better judged on how long it took. Either way the metric was declared up front, and the outcome table supplies whatever was actually measured.
There is a floor under all of this that randomization cannot reach. zcl_outcomes.success is a bare BOOLEAN NOT NULL, and nothing in the schema says who decides it or on what evidence. Randomizing the assignment removes the confound between the user and the branch they got. It does nothing whatsoever about a mismeasured outcome. If success is set by a heuristic, or reported by the same person whose skill I was trying to control for in the first place, then the bias I built this entire layer to remove walks back in through the dependent variable — and this time the system reports it with a p-value attached, which makes it harder to argue with rather than easier.
If I want to judge some other behavior later, the outcome schema has to carry it first, or the candidate cannot be promoted inside this loop at all. It is a narrow door to walk through, and I would still rather have it than a soft win assembled from whatever trace happened to be lying nearby.
6. Minimum samples slow the system on purpose
ActiveLearning sets alpha = 0.05, with the code comment p < 0.05 for significance. The result object carries effect_size, p_value, significant, and recommendation. Evaluation compares the two arms' success counts with a chi-square test on a two-by-two contingency table, and the migration includes experiment_ready_for_analysis, which checks whether an experiment has enough samples before evaluation.
This is the guardrail I wanted most. A document can land in treatment, ride along with one successful session, and look useful if the system is hungry for reinforcement, so the threshold is supposed to make it wait.
At least that is what I had been telling myself. Writing this section I went to check the number, and the number is not wired to anything.
Hypothesis.min_samples is set to 20. Nothing reads it. It sits on the dataclass, rides into the row, and not one code path consults it before a result gets judged. The gate that actually runs is in analyze_experiment, which bails out only when an arm holds fewer than five outcomes, and its SQL counterpart experiment_ready_for_analysis takes p_min_per_group INTEGER DEFAULT 5. So the brake is five, not twenty. The twenty is decoration, and it is the kind that looks clean and fails quietly.
Five per arm cannot carry the conclusion the code is willing to draw from it. Work the table: five sessions in each branch, chi-square at alpha 0.05, and exactly two of the thirty-six possible outcomes clear significance. Five successes against zero. Zero against five. Both land at p equal to 0.011, and every other cell in that table is a null before the experiment starts. At this size the test is not measuring an effect. It is asking whether the two arms disagreed about every single session, which is not the question I meant to ask.
Chi-square is the wrong instrument here anyway. It wants expected cell counts of about five or more, and a two-by-two table built from five observations per arm cannot give it that. Fisher's exact test handles tables this small and the swap is one line. Raising the gate to the twenty already written on the object does not rescue it either — at twenty per arm the smallest difference the test can see is still around thirty percentage points.
Here is the scale I should have worked out before writing any of it. A ten point improvement in success rate, 0.50 to 0.60, at eighty percent power and the same alpha, needs roughly 194 sessions per arm. Five points needs about 782. Those are the numbers at which the words sitting in recommendation — ADOPT, REJECT — mean what they claim. Ten points is a substantial win for a context change. This layer would need forty times the evidence it currently demands to notice one.
There is a related exposure in how many tests run at once. The generator can open five order hypotheses and three inclusion hypotheses for a single task type. Eight independent tests at alpha 0.05 carry roughly a 34 percent chance that at least one clears significance by luck alone, and the one that clears is the one that gets promoted into every future session's context. A Bonferroni correction, or simply refusing to run more than one live experiment per task type at a time, is the cheap fix.
One thing the design gets right by accident of structure is worth crediting, because it is the error most A/B systems make. analyze_experiment completes the experiment as soon as it evaluates it. There is no path that looks at the data, finds nothing, and looks again next week. Repeated peeking at an accumulating result is how a nominal five percent false positive rate becomes twenty or thirty percent in practice, and this loop cannot do it: one look, then the experiment closes. The cost of that virtue is that the single look happens at the earliest moment it is permitted, which is also the moment with the least evidence behind it.
Put those two facts side by side and it is worse than slow learning. A completed experiment never reopens; _complete_experiment writes status completed and nothing anywhere sets it back to active. At five per arm almost every outcome is INCONCLUSIVE by construction. So a context change that genuinely helps gets one underpowered look, comes back as no significant difference because it could hardly come back as anything else, and is closed on that basis permanently. I built the threshold to stop the system adopting noise. What it actually does is retire real improvements the test was never equipped to detect, and there is no route back to retry one.
The exploration rate decides where this lands hardest, and it picks the worst place. _get_task_uncertainty returns maximum uncertainty for any task type with fewer than ten recorded sessions, which pushes the exploration probability straight to its 0.4 ceiling, and that count is scoped per organization. A small organization therefore runs the largest share of its sessions as trials while generating the fewest sessions to finish any of them with. It experiments hardest precisely where five outcomes per arm takes longest to accumulate, and every one of those trials still closes after a single look.
The stored result hides that rather than surfacing it. effect_size goes into the results JSON as a bare difference between two success rates, with no interval around it, so a gap measured on five sessions per arm is recorded in the same shape, and reads later with the same authority, as one measured on five hundred.
Waiting costs something. Bad ideas stay alive longer, good ones take more sessions before they become default behavior, and some combinations never get tested at all because the generator cut them off early. That pressure is almost certainly how the number ended up at five: a gate that low returns verdicts fast, and returning verdicts fast is the entire problem with it.
The operational states are active, completed and cancelled. An active experiment can still receive assignments and a completed one can store results, but the useful one is cancelled: it stops shaping sessions without pretending it ever reached a conclusion. The view v_experiment_results puts treatment counts, control counts, status, and timestamps together so the operator can see whether a run is still gathering evidence or ready for judgment.
That is the trade I made for agent memory and retrieval: learn slower, record the branch, measure after work, and require enough samples before future sessions change. Three of those four are built and working. The fourth is a number I wrote on an object and never wired up, and writing this post is what made me look.
The rule I started with still holds. A learning system that changes future inputs has to earn that change with outcomes, or it will preserve coincidence as policy. Randomization is what makes an outcome mean anything. The sample threshold is what stops it being noise. I built the first one properly. The second I wrote down on a dataclass and never wired up, so the system will go on drawing conclusions from five sessions and filing them with the same confidence it would give five hundred.
🎧 Listen to the audiobook — Spotify · Google Play · All platforms
🎬 Watch the visual overviews on YouTube
📖 Read the full 13-part series
