My drift detector quarantined a healthy agent. The agent hadn't gotten worse — the benchmark had gotten bigger. The detector compared failure counts across corpus versions, so growing the benchmark from 10 tasks to 20 made a 10/10 agent look like it "failed" half the suite. Corpus growth looked like decay.
I built the drift detector for HivePlane v0.2.0 to catch the failure mode I'd written about in v0.1.0 — "the dangerous agent usually drifted, not got edited." The harder problem turned out not to be detecting drift. It was not detecting drift that wasn't there. A detector that cries wolf trains operators to ignore the signal, and an ignored gate is no gate.
The detector that ate its own benchmark
| Finding | What the detector did | Why it was wrong |
|---|---|---|
| #530 | Compared failure counts across corpus versions | Growing the benchmark auto-quarantined a healthy agent |
| F-13 | Reinstatement required status == QUARANTINED
|
Auto-quarantine resets to UNCERTIFIED — the gate refused its own quarantine |
| #609 / S24 | The probe runner only checked admission | A probe could never flag behavioral decay — only a refused run could fail one |
The scariest line in the release wasn't a missed drift. It was a healthy agent quarantined because the detector compared it to a benchmark that no longer existed.
The benchmark that looked like decay
530 was the one that changed how I think about drift baselines. The detector compared failure counts across corpus versions. If the benchmark grew from 10 tasks to 20 — because the learning loop added a reviewed case — a healthy agent that passed 10/10 now "failed" 10 of 20 by the old accounting. Corpus growth looked like decay.
The agent hadn't gotten worse. The benchmark had gotten bigger. The detector couldn't tell the difference, so it quarantined a perfectly good agent.
The fix: drift is measured against the agent's own baseline on comparable corpora. You compare the same tasks across re-evaluations, not different counts. A corpus version change establishes a new baseline, not a drift signal. Growing the benchmark is growth, not regression.
tip: If your drift metric is a count, a benchmark change will look like behavior change. Compare the agent to itself on the same tasks, or any corpus edit becomes a false quarantine.
The reinstatement that refused its own quarantine
F-13 was the symmetry that broke. Drift auto-quarantine resets the certification status to UNCERTIFIED — not QUARANTINED. Reinstatement required status == QUARANTINED. So the gate that created the quarantine refused to lift it. A workload could be auto-quarantined and then never reinstated, because the precondition described a state the detector never set.
The fix was removing the stale precondition. An active quarantine record is reinstatable — the workload is re-certified and admission is re-checked. The status field is not the source of truth; the quarantine record is. The gate keyed off the wrong one.
The field test caught this in scenario S5, where reinstatement after S4's auto-quarantine failed in sequence. Two scenarios, one bug, and the lesson: the source of truth for "is this quarantined" is the quarantine record, not the certification status enum.
The probe that couldn't smell decay
609 / S24 was the most subtle. The synthetic probe runner ignored Spec.expected and only checked admission. So a probe could only fail if the run was refused — it could never flag that the agent ran and produced the wrong answer. The probe was a liveness check pretending to be a readiness check.
A probe's job is to catch decay before the drift threshold trips. If the probe only checks "did it run," it can't catch "did it run correctly." The fix: the runner starts the run, waits for terminal, and compares the output to expected. A mismatch fails the probe with a probe.decay warning. Now a probe can smell decay the drift detector hasn't measured yet.
What I learned
False-positive controls are the product, not a nicety: a single exceeding run is a warning, and quarantine requires N consecutive failures or a strong signal — a stable agent is never quarantined on noise. Measure drift against comparable baselines: the same tasks, the same corpus version; a corpus bump is a new baseline, not a regression. The quarantine record is the source of truth, not the status enum — reinstatement keys off the record, or it refuses the very quarantines it should lift. And a probe must assert the expected output, not just that the run was admitted — liveness is not readiness.
The v0.2.0 field test demonstrates the full loop — drift due/assess/probe, auto-quarantine, reinstatement after a fresh certification — as scenarios S4, S5, S8, and S24. The synthetic probe flags decay before the drift threshold in S24. Full evidence in the field-test report; the certification and learning-loop designs are in the certification v2 design and the learning loop design.
References
-
HivePlane —
pip install hiveplane==0.2.0 - Field test report (36/36)
- Certification v2 design · Learning loop design
- Operator runbook — drift & quarantine
When was the last time your detector quarantined something that was fine? If your operators have learned to reinstate without reading, your detector isn't a control — it's a checkbox.
