A lost update between AI agents: two agents edit the same resource, both writes return success, and one is silently gone with nothing in the system saying a word. It happens when both read the same version and the last writer overwrites the rest. The fix is a pre-write compare-and-set gate, not a bigger log.
I ran it. With 5 agents committing to one shared file under a worst-case schedule, where every agent reads before anyone writes, 4 of the 5 contributions vanished from the final state, and every one of those 5 writes had been acknowledged. The final file held exactly one agent's work. The other four were paid for, ACKed, and silently overwritten. And that worst case is not rare: in a fair sample of interleavings at N=5, some write was lost in essentially 100% of runs.
This is the lost update anomaly, and it is older than AI agents by decades. What is new is that each vanished write burned real model tokens, and the loss leaves no error, no exception, no red log line. So I built a small offline simulator to make the loss countable, and to test the one class of fix that actually stops it: a version check before the write lands.
AI disclosure: I wrote
lostwrite_sim.pywith an AI assistant and ran it myself, offline, on Python 3.13.5, standard library only. No network, no keys, no threads, no funds. Concurrency is simulated by explicit interleaving schedules, not real parallelism, so the output is reproducible. Every number and hex string below is pasted from a real local run. Three runs produced byte-identical STDOUT with sha2561df08c5e38894622314a2b052684303853b9b8fa9508eaabdfd877970c88983b. The linked specs and articles are other people's work.
In short:
- When N agents read the same version of a resource, each does its work, and each writes back, the last writer wins and everyone in between is silently overwritten. In the worst case at N=5 that was 4 lost writes out of 5, all acknowledged.
- Whether it happens at all depends on the schedule. In a fair sample of interleavings the loss hit 75.0% of runs at N=2 and effectively 100% by N=5, but a quarter of N=2 runs serialized and lost nothing. So one green test run proves nothing.
- An append-only log records every write, so you can prove each write happened. It does not help you recover one. Reading the current value still returns a lost update, because logging preserves the fact of a write, not the presence of its change.
- A pre-write compare-and-set (check the version has not moved since you read it) drops lost updates to 0 by construction. Measured, it lost nothing in all 46,333 fair interleavings I sampled, at every N.
- The gate does not save money. It costs more. In the worst case at N=5 it added 4 work-units of retries on top of 5, so total work went from 5 to 9. It buys integrity and a countable collision, not a refund.
- Turn the version check off inside the same gate and the loss comes straight back (4 again). That falsifier is the proof that the compare, not the scaffold around it, is doing the work.
This is not a toy concern anymore. The moment you run two sub-agents, or a swarm, or a retrying planner that spawns workers, you have multiple actors writing shared state: a plan file, a scratchpad, a database row, a GitHub issue. @opsveritas wrote up the paired version of this: one agent times out, three more do not notice, and work gets paid for that no one will ever read. Lost update is the same failure seen from the write side.
What is a lost update between two AI agents?
A lost update happens when two actors read the same version of a resource, both compute a new value from what they read, and both write it back. The second write is based on a snapshot that predates the first write, so committing it erases the first. Neither actor sees an error. The classic definition lives in database isolation theory, and agents inherit it the instant they share a resource without a write precondition.
The mental model I used in the simulator is deliberately concrete: N agents each append their own section to one shared file. The correct final file contains all N sections. A lost update is a section that made it into an acknowledged write and then disappeared from the final file.
Here is the store. It holds a version number and the set of contributions currently present.
class VersionedStore:
def __init__(self):
self.version = 0
self.value = frozenset()
self.wal = [] # append-only log of every accepted write
def write_lww(self, agent_id, new_value):
"""Last-writer-wins: overwrite unconditionally, always ACK."""
self.value = frozenset(new_value)
self.version += 1
self.wal.append((self.version, agent_id))
return "ACK"
def write_cas(self, expected_version, agent_id, new_value, enforce=True):
"""Commit only if the version has not moved since this agent read."""
if enforce and self.version != expected_version:
return "CAS_FAIL"
self.value = frozenset(new_value)
self.version += 1
self.wal.append((self.version, agent_id))
return "ACK"
Each agent runs three steps: READ the store, WORK (build snapshot | {my_id} and spend one unit), then WRITE. Concurrency is not threads. It is an explicit interleaving, and I run it two ways on purpose. The first is a deterministic worst case: every agent reads before anyone writes, so every read is stale. That is the adversarial upper bound, not a typical run, and I use it for the headline because it is the clearest picture of the failure. The second is a fair sampler: at each tick pick one still-ready agent uniformly at random (random.Random, seeded, so the whole run is reproducible to the byte), and draw tens of thousands of independent interleavings to get a distribution. The worst case shows the failure; the fair sample shows how often it actually bites.
A note on that, because the first version of this tool got it wrong and it matters. I originally picked agents with a hand-rolled linear congruential generator and next() % n, which reads the least-random low bits. At N=2 that strictly alternates for every seed, so it silently pinned the worst case and printed it as if it were a seeded sample. random.Random.randrange draws from the full generator state and does not have that bias. The fair numbers below come from the fixed version.
One thing I want to be honest about up front, because it decides how you read every number here. The costs in this tool are counts, not dollars. COST_UNIT is an accounting constant set to 1, labelled in the source as "not dollars, not a measured price." Every money figure is an integer counter times that constant. If you want your dollars, multiply the run counts by your own measured per-run token cost. I did not measure a token price, and I am not going to pretend a synthetic schedule tells you one.
The anomaly at its worst: N=5
5 agents, the worst-case schedule (everyone reads before anyone writes), gate off versus gate on. This is the headline block straight from the run:
WORST CASE N=5, maximum contention (every agent reads stale)
--------------------------------------------------------------------
NO_GATE (LWW) GATE (compare-and-set)
writes acknowledged 5 5
contributions in final 1 5
contributions expected 5 5
lost_updates (ACKed, gone) 4 0
integrity_ok (present==all) False True
cas_failures 0 4
retries 0 4
refusals 0 0
total_runs (work executed) 5 9
total_cost (runs*COST_UNIT) 5 9
Read the left column first. Five writes acknowledged, one contribution in the final state, four lost. The lost_updates number is not printed from a flag that says "no gate, so print 4." It is re-derived: the tool takes the set of agents whose write was ACKed at least once, subtracts the set actually present in the final value, and counts what is left. Four writes said success and are not in the file.
The right column is the same schedule with a version check on the write. Zero lost. All five contributions present. And I need to be blunt about why that zero is a zero, because it would be dishonest to sell it as a surprising discovery: it is true by construction. A compare-and-set write only commits when the version has not moved, which means it commits on top of a snapshot that already contains every prior commit. It cannot drop one. The interesting, measured part of the right column is not the zero. It is the price of the zero, which I will get to.
Tracking is not control
Here is the part I keep having to relearn. "Just log every write" feels like it should help. It does not.
TRACKING IS NOT CONTROL (NO_GATE, worst case, N=5)
--------------------------------------------------------------------
writes recorded in append-only log (WAL) : 5
WAL entry shape : (version, agent_id), no value
contributions surviving in final state : 1
final state contains contributions : (4,)
writes logged and ACKed but gone on read : 4
The append-only log has all five writes. Every one. It proves that five agents each wrote, and it tells you which agent and at what version. That is enough to know you were charged for five and kept one. It is not enough to hand any of the four back: my WAL entry is (version, agent_id), the fact of the write, not the value it carried. A richer log that stores the content, like the separate git-refs @dipankar_sarkar described for two coding agents editing the same issue, keeps both versions and reconciles after, and for a merge-later workflow it is the right tool.
But look at the last line, and notice it holds no matter how fat the log gets. The log knows about all five writes, and reading the current value still returns four lost updates. Recording a write and preventing its loss are different operations at different times. The log is a receipt written after the fact. It cannot un-overwrite the file, any more than a receipt read after the fact can recover a field that was never stored. This is the same franchise as a spend cap that counts tokens but cannot stop the loop: tracking tells you what happened, control changes what is allowed to happen. Only a write that runs before the commit gets to reject the stale one.
The gate, and its honest price
The fix is a precondition on the write. Optimistic concurrency, the lock-free version: read a version, do your work, and at commit time refuse if the version moved. It has a formal name, optimistic concurrency control, and the primitive under it is compare-and-swap. A pessimistic lease or lock would also work; it just makes the agent wait instead of retry. Either way the shared idea is the same: no write lands on state you did not read.
res = store.write_cas(a.base, a.aid, a.newval, enforce=not disable_version_check)
if res == "ACK":
a.acked = True
else:
cas_failures += 1
if a.retries < max_retries:
a.retries += 1
a.pc = "READ" # re-read the latest, re-work, re-write
else:
a.refused = True # fail-closed: nothing silently applied
Now the cost, which is the whole reason this post is not called "how the gate saves you money." It does not. Look again at the headline: NO_GATE ran 5 work-units, GATE ran 9. The gate added 4 work-units of retries, because each stale writer had to re-read fresh state and redo its work before it could land. The delta is positive, and it is positive by design:
Delta cost of the gate = 9 - 5 = 4 work-units (>=0: the gate
costs MORE, it does not save tokens)
Spend destroyed without gate= lost_updates * COST_UNIT = 4 work-units
(already paid, silently gone from state)
Two numbers, two different things, and it is easy to blur them into a false savings pitch. The 4 destroyed work-units under NO_GATE are money you already spent on work that then vanished. The gate does not refund them. It cannot reach back into a run that already happened. What it does is stop the next four from vanishing, and it charges you 4 units of retries to do it. If your worry is the token bill, the gate is a cost, not a saving. If your worry is that four agents did real work and the result quietly disappeared, the gate is the thing that keeps that from being silent.
That is the contrarian bit, and I will state it as a falsifiable claim: a pre-write concurrency gate on shared agent state increases token spend and is still worth it, because it converts a silent unbounded loss into a visible, counted, retryable event. If your data shows a gate that reduces total spend under contention, I would genuinely like to see the workload, because my own run says the opposite.
Is the gate doing the work, or is my test scaffold cheating?
This is the failure mode I trust least in myself, so the tool has a falsifier for it. The classic mistake is to build a "gate" whose zero comes from the test scaffold rather than the mechanism. So F1 keeps the entire gate code path and disables one thing: the version check inside write_cas.
FALSIFIER F1 gate code path, version-check DISABLED (worst case, N=5)
--------------------------------------------------------------------
lost_updates with the compare removed : 4 (was 0 with it on)
Four again. Same as no gate. The retry loop, the counters, the refusal path, all still there. Remove only the comparison of expected version to current version, and the loss returns in full. That is the proof I actually care about: the compare is the mechanism. Everything else is plumbing.
Does it get worse with more agents?
Two questions live here, and they have different answers. How bad can it get, and how often does it get bad at all. The worst case answers the first. The fair sample answers the second, and it is the more useful number.
First the worst case, swept over N. Every agent reads before anyone writes, so all but the last writer are clobbered. The N=1 row is the negative control: one writer, no one to overwrite, zero loss.
WORST-CASE SWEEP over N (maximum contention, every read stale)
----------------------------------------------------------------------------
N | ng.lost ng.integ | g.casfail g.retry g.refuse g.lost g.integ | dCost
--------------------------------------------------------------------------
1 | 0 True | 0 0 0 0 True | 0 (degenerate: 1 writer)
2 | 1 False | 1 1 0 0 True | 1
3 | 2 False | 2 2 0 0 True | 2
5 | 4 False | 4 4 0 0 True | 4
8 | 7 False | 7 7 0 0 True | 7
In the worst case the loss is exactly N-1: everyone but the last writer is gone, and the gate pays one retry per clobbered writer to fix it. But nobody schedules their agents to lose. The honest question is what a fair draw of interleavings does, so I sampled 40000 // N of them per N and counted the distribution.
FAIR SWEEP uniform random interleavings (random.Random, Mersenne Twister)
----------------------------------------------------------------------------
base_seed=20260728, trials per N = 40000 // N
N | trials | NO_GATE loses>=1 | serializes (0 lost) | mean lost | max | GATE loses>=1
--------------------------------------------------------------------------
2 | 20000 | 75.0% (14994) | 25.0% ( 5006) | 0.75 | 1 | 0
3 | 13333 | 97.3% (12972) | 2.7% ( 361) | 1.59 | 2 | 0
5 | 8000 | 100.0% ( 7999) | 0.0% ( 1) | 3.35 | 4 | 0
8 | 5000 | 100.0% ( 5000) | 0.0% ( 0) | 6.14 | 7 | 0
This is the number I would actually put in a design doc. At N=2 a quarter of interleavings serialize on their own and lose nothing, which is exactly the trap: two agents, run it a few times, watch it pass, ship it. Then it loses data in three runs out of four. By N=3 the safe fraction is down to 2.7%, and by N=5 it is one interleaving in eight thousand. So the anomaly is not universal on small N, and it is close to certain past a handful of writers. That is a worse story than "it always happens," not a better one, because "usually fine" is the thing that gets shipped.
The gate column is the point of the whole exercise. Across all 46,333 fair interleavings I drew, over every N, compare-and-set lost an update exactly zero times. That is a measured claim, not a hand-wave: the by-construction argument says a CAS write cannot drop a prior commit, and 46,333 independent draws agree. NO_GATE integrity is False on every row with a second writer; GATE integrity is True on every one of them; and the gate's delta cost is never negative.
When the gate runs out of retries
A gate that only works when it can always retry is not much of a gate. So the last thing I checked is the ugly case: heavy contention with zero retry budget. What does it do when it cannot win the race?
FAIL-CLOSED GATE with max_retries=0 (worst case, N=8)
--------------------------------------------------------------------
cas_failures : 7
refusals (VISIBLE, counted) : 7
refused agent ids : (1, 2, 3, 4, 5, 6, 7)
lost_updates (SILENT) : 0
integrity_ok : False
Seven agents lost the race and, out of retries, they refused. Loudly. integrity_ok is False, which is correct, because seven contributions genuinely did not make it. But lost_updates is 0. Nothing was silently overwritten. The difference between this and the NO_GATE case is the entire point: NO_GATE gives you integrity_ok=False with a silent 4, and you find out never. The gate gives you integrity_ok=False with seven refusals by explicit id, and you find out immediately. One failure is invisible, the other is a list of names. That is what "fail-closed" buys, and the tool exits non-zero if that property ever breaks.
What this is, and what it is not
It is a simulator, not a measurement of production. Two kinds of number live in it and they deserve different trust. The worst-case counts (headline 4 lost, N-1 across the sweep) are a deterministic upper bound: they are what a maximally adversarial schedule does, not what a typical one does, and I label them that way. The fair-sweep fractions (75.0% at N=2, 97.3% at N=3, ~100% at N=5 and N=8) are Monte Carlo estimates over tens of thousands of uniform-random interleavings, so they carry the usual sampling error of a proportion and would shift a little on a different base seed. What does not shift is the direction: the zero-loss fraction falls hard as writers are added, and the gate's zero holds in every sampled interleaving. None of this is a token-price measurement. I sampled one model of "random interleaving," a uniform scheduler; a real runtime with its own scheduling could sit anywhere between my worst case and my fair sample, and I have not measured a real one.
I also did not measure a token price, and I will not dress the accounting unit up as one. And this is not the revert-guard problem, where a single agent reintroduces reverted code, nor is it about a single agent double-charging on a retry. It is specifically two-or-more writers, one version, no precondition. If you run sub-agent dispatch or any fan-out where workers share a resource, this is the write-side gate that belongs next to your pre-execution checks. And it is a distinct axis from mandate freshness: freshness asks whether an old approval is still valid in time, this asks whether two live writes can both survive.
Run it yourself
Standard library only, offline, no keys, no funds, a few seconds (the fair sweep draws about 92,000 interleavings). run_all.sh runs the selftest, then three full runs, compares them byte for byte, and prints the sha256 of each.
interpreter: Python 3.13.5
self-test: PASS
run 1: exit=0 sha256=1df08c5e38894622314a2b052684303853b9b8fa9508eaabdfd877970c88983b
run 2: exit=0 sha256=1df08c5e38894622314a2b052684303853b9b8fa9508eaabdfd877970c88983b
run 3: exit=0 sha256=1df08c5e38894622314a2b052684303853b9b8fa9508eaabdfd877970c88983b
determinism: 3 runs byte-identical
The report body also self-reports its own sha256 (57e02acaa779e066d4736bb00b9ef1a2f4eeec21d886c626879cfd1a67db661e), so you can tell at a glance whether your run matches mine. The selftest is the contract: INV1 requires the worst case to lose exactly N-1, INV2 requires the gate to lose nothing there, INV3 requires the fair sample to lose in some but not all interleavings on small N, INV4 requires the gate to lose nothing in any sampled interleaving, F1 through F4 are the falsifiers, and any failed assertion exits 1 with FAIL. A gate that cannot fail its own tests is decoration.
Follow along if you want the numbers from the next teardown in this series. And if you are running more than one agent against shared state right now, tell me in the comments: what is the worst silent overwrite you have hit between two agents, and did you catch it before the commit or only when the result was already gone? I suspect, for most of us, the honest answer is "only when it was already gone."
