Observability for multi-agent systems: instrumenting for split-brain, goal drift, and context drift

Originally published on Loop & Retry — field notes on building LLM agents that survive production.

Diagnosis without instrumentation is just vocabulary. This cluster has now named three ways a crew of agents breaks that no single agent can break by itself — split-brain, goal drift, and context drift — on top of the four in failure modes in multi-agent teams, and each of those posts ends the same way: go check. This one is about what "go check" actually costs to build, where the probe lives, and which parts of it you can afford to run continuously.

Per-agent monitoring is structurally blind, not just insufficient

The usual setup is per-agent: each agent's run emits a trace, a token count, a latency, a pass/fail on its own acceptance criteria, maybe a transcript to pull up when something goes wrong. It catches everything that goes wrong inside one agent — the malformed tool call, the blown latency budget, the loop that never terminates.

It catches none of this cluster, and the reason is worth being precise about, because "better monitoring" is the wrong conclusion. Look at what each failure actually is:

• Split-brain is two agents holding different values for one key.
• Goal drift is the current hop's spec differing from hop zero's.
• Context drift is one agent's stated beliefs differing from the canonical objective's current version.
• Correlated collapse, from the anchor post, is high inter-agent agreement paired with mediocre conditional accuracy — also a cross-agent quantity.

Every one is a diff between two records — not a threshold on one record, not an exception, not an anomalous value you could alert on. A per-agent telemetry store cannot express that diff no matter how complete it is per agent, because it's indexed by agent-run. To ask "did these two agents disagree about key K inside task window W," you have to group by (task, key) and compare across agent — and if those aren't fields your telemetry carries, the query isn't slow, it's unwritable.

This is the multi-agent form of the problem a transcript is a log, not an index describes for one agent: stop storing prose you have to re-read, start storing keyed events you can query. Same move, one level up — the missing primitive isn't a metric or a dashboard, it's a join key. Every record an agent emits has to carry the task id that scopes the comparison, the id of the agent emitting it, and a version or timestamp for whatever it read. Those three fields are the whole difference between a coordination failure being detectable and being invisible.

And those fields have to be on the write path. By the time you want to ask the question, the run is over — standard late-detection economics, where the instrumentation exists before the incident or the incident is unanalyzable.

Split-brain: instrument the store, not the agents

Split-brain's diff is over reads and writes of shared state, so the probe belongs in the store client — a thin wrapper every agent goes through — and not in any agent's own logic. Agents will not reliably self-report; a wrapper has no choice.

The wrapper's job on every access is to emit five fields: (task_id, agent_id, key, value_hash, store_version). That's it. From that stream, two checks fall out almost for free:

The divergent-reads check. Group reads by (task_id, key) and count distinct value_hash values. A group with more than one means two agents in the same task were working from different values of the same fact. That is split-brain, observed directly rather than inferred, and it's a GROUP BY — no model call, no sampling, no heuristic. Run it continuously and alert on distinct_hashes > 1.

Write-after-stale-read. Every write carries the store_version its source read came from. Compare that against the store's version at commit time. A write whose source version is behind is a write built on a superseded baseline, whether or not it succeeded. Rejecting those writes is optimistic concurrency — that's the fix. Counting them is the observability, and you want the count even after the fix ships: a rising stale-write rate says contention is growing before it produces an incident.

The trap: a value hash is only as good as its canonicalization. Two agents reading semantically identical state that serializes with different key ordering or float formatting will trip divergence alerts on every read, and the check gets muted within a week — worse than not having it. Canonicalize before hashing.

Goal drift: instrument distance from origin

Goal drift's diff is between hop zero and hop N, so its probe attaches to specs and handoffs, not state. It needs one artifact created at hop zero and carried forward: a constraint ledger — the original ask's non-negotiables, extracted once, stored verbatim, one stable id each.

Constraint survival is then a set operation, and it's the one to build first. Each downstream spec declares which constraint ids it carries. Survival rate is |constraints present at hop N| / |constraints at hop zero|, per hop, and the useful output isn't the ratio — it's the named missing ids. "Constraint C3 was last seen at hop 2" points at the exact handoff that dropped it. It's string matching against a fixed list, so it's free on every hop, and it degrades gracefully: an untagged spec scores zero and surfaces as a coverage gap rather than a silent pass.

Distance from origin is the softer one: embed the current working spec and the original ask, track cosine distance across hops. Two cautions, because it's easy to deploy as noise. The absolute number means nothing — legitimate decomposition also moves a spec away from the original ask, since a subtask should read differently from the whole request. You're watching the slope: a monotonic increase across hops, never a single reading. Which means it needs a baseline of healthy runs to be interpretable, so treat it as a trend you review, not a threshold you page on. Constraint survival is the alerting metric; distance from origin tells you which runs to go read.

Context drift: instrument refresh age first, beliefs second

Context drift's diff is between what an agent currently believes about a fixed objective and what that objective now says. Comparing beliefs means asking agents to state them, which costs a model call each — so the discipline is a two-tier probe, cheap tier always on.

Tier one: refresh age. Timestamp every agent's last objective refresh; timestamp every correction issued, along with the set of agents holding a copy of what it corrected. Any agent whose refresh timestamp predates a correction in its holder set is provably stale. Pure metadata arithmetic — no model call, no sampling, no judgment — and it catches the whole class of failure where the update never went out. Two derived numbers worth emitting: correction fan-out (what fraction of the holder set acknowledged a refresh) and max staleness (the oldest refresh age among active agents). A fan-out below 100% is the bug, not a rounding error.

Tier two: the belief poll. Refresh age catches missed broadcasts but not the case where an agent was told and absorbed it differently. For that you have to ask: at a checkpoint, prompt an agent to restate the objective's current constraints in one sentence, and compare.

Compare against what, though, matters more than it looks. The instinct is to diff agents against each other, which is n(n-1)/2 comparisons and gives you divergence with no indication of which side is right. Diff each agent against the canonical objective statement instead: that's n comparisons, it identifies which specific agents are wrong rather than just that someone is, and it's sufficient — if every agent matches canonical, every pair agrees.

And keep the comparison narrow. Asking a model "do these two statements agree?" is exactly the open-ended judging that an LLM judge is least reliable at — it will find agreement in paraphrase and disagreement in phrasing. Give it a closed task instead: here is the canonical constraint list by id, here is the agent's restatement, return which ids the restatement asserts, which it contradicts, and which it omits. That's extraction against a fixed schema, gradeable and stable, rather than a vibe check on semantic equivalence.

Implementation: what to emit, and how to sample

Six of the seven are metadata you can leave on permanently. One costs real money, and its sampling policy determines whether the whole scheme is affordable. Four rules:

1. Trigger on state change, not on wall-clock. Divergence is created by events — a correction issued, a handoff completed, a hop entered. Polling every five minutes spends most of its budget confirming nothing happened since the last poll. Poll after the events that create it.
2. Let tier one pick the sample. Refresh age is free and already ranks agents by likelihood of being stale. The candidate set isn't "all active agents," it's "agents whose refresh predates a correction in their holder set" — usually small, often empty, and empty means you skip the checkpoint at zero cost. Biggest lever here, and the reason to build tier one first.
3. Stratify by blast radius, not uniformly. If the candidate set is still too big, sample the agents whose output leaves the system — the responder that talks to the customer, the executor that writes to prod. An intermediate agent's stale belief usually gets a second look downstream; the last agent in the chain has nothing after it. Uniform sampling treats those as equal; they aren't.
4. Budget the probe like a retry budget. Cap belief-poll calls per task, make hitting the cap a visible outcome rather than a silent stop, and account for them explicitly — observability calls are part of what a run actually costs, and an unbudgeted probe that scales with crew size is how a monitoring layer quietly becomes a line item nobody approved.

Build order follows the cost column: join keys first, because nothing works without them; then the free probes; then the sampled belief poll, aimed by refresh age at the agents already provably stale.

The summary

Multi-agent observability is not per-agent observability with more dashboards. It's the same shift distributed systems made when they stopped grepping per-process logs and started propagating a trace context: the failures worth catching are relations between records, so the records need the fields that let you relate them. Get task_id, agent_id, and a version stamp onto every read, write, spec, and refresh, and split-brain becomes a GROUP BY, goal drift becomes a set difference, and context drift becomes a timestamp comparison — three of the cluster's failure modes rendered queryable by metadata alone, before any model call enters the picture. Skip those fields and no amount of per-agent monitoring will surface any of them, because each agent's own telemetry reads as clean — the property all of these failures share, and the reason measuring the whole trajectory has to mean the whole crew's, not each agent's separately. If you're running a crew today and can't write the query "did any two agents in this task disagree about the same key," that's not a gap in your dashboards. It's a missing column, and it's on the write path.