Your Agent Has a 1987 Database Problem

Here is the shape of a bug I have watched people chase for days.

14:22:03.204  tool    POST /v1/charges              201 Created
14:22:03.209  db      COMMIT orders/8812 status=paid
14:22:03.211  x       process killed (out of memory)
14:22:11.470  worker  resume th_44c1 from ckpt_19
14:22:11.882  graph   node=tools    call_7f3a has no result
14:22:11.955  tool    POST /v1/charges              201 Created

The customer was charged twice, and nothing in that trace failed. No exception, no retry counter, no red span in the viewer. Both charges returned clean 201s, and the agent did exactly what it was built to do.

So why did it charge twice? The answer sits one layer down, in how an agent loop records what it has already done.

How an agent loop actually works

A model has no memory. Calling one is a pure function call. It does not remember the previous call, holds no connection to your database, and has no idea whether it charged someone thirty seconds ago. Everything it knows arrives as input, and the moment the call returns it knows nothing again.

So the agent's memory is a list of messages your runtime owns. The system prompt, the user's message, each assistant reply, each tool requested, each result handed back. That list lives in Postgres and gets passed to the model on every turn. When people say an agent knows something, this list is what they mean, because there is nowhere else for knowledge to live.

One turn contains two separate writes. Your code executes the tool, which moves money at a payment provider and writes a row in your own database. Then it appends the result to the list, and the runtime saves the list. Executing and saving happen at different moments.

In the gap between them, the agent's memory disagrees with the world. The charge exists permanently at the provider. The list does not mention it.

The rest of this article is about that gap.

Whether it matters to you depends on what your tools actually do. An agent that searches, summarizes or drafts can crash and replay all day without harm, because running a read a second time costs latency and nothing else. The gap only bites when a tool changes something outside your process that you cannot take back, meaning money moving, an email going out, a ticket filed, a row written into somebody else's system. If your agent only reads, none of what follows is urgent. If it acts, all of it is.

What happened in the trace

Two questions to answer: where was the gap, and what was in the checkpoint when the process died?

LangGraph runs your agent as a graph. One node calls the model, a second node executes whatever tools the model asked for, and after each of them a component called the checkpointer saves the message list.

The checkpointer is the save function, and nothing more exotic than that. It writes the agent's current state to storage, keyed by conversation, so a different process can pick the run up later. In development it is usually an in-memory stub; in production it is Postgres. Each save it makes is a checkpoint, and ckpt_19 in the trace is one of them.

At ckpt_19, that saved list ended like this:

assistant   "I'll charge the customer."
            tool_calls: [ charge_customer(cus_8812, 4900), id=call_7f3a ]

That is a question with no answer: a tool was requested, nothing has come back, and the graph's next move is to run the tools node.

The tools node then did four things in order:

Called the payment provider.
Took the 201 back.
Committed the order row.
Built the reply message that would close out call_7f3a.

It was killed before it handed that reply back. Steps one and three are permanent and outside your control now, because the money moved and the order row committed. Steps two and four existed only in the memory of a process that no longer exists.

So the checkpointer was never given anything to save, and ckpt_19 still ends exactly as it did above, with a tool call and no result.

When the worker resumed eight seconds later it loaded that list and did the only thing available to it. An unanswered tool call means run the tool, so it ran the tool.

Nothing malfunctioned. The graph followed its own rules against a record that was missing an entry, and in LangGraph this is documented behavior rather than a bug, because saving the list is a step that comes after the tool has already done its work.

So the question is not why the agent repeated itself, since given that list it had no other option. The question is why the charge and the record of the charge were allowed to come apart at all. Two things needed to happen together and did not, which is a problem databases have been solving since 1978.

1978: all or nothing

Move fifty dollars between two accounts and you have two writes: subtract from one, add to the other. If the machine dies in between, the money is not queued somewhere, it is gone, and nothing in the system knows it existed. The transaction had to be invented to stop this. Until it existed, an application that needed two writes to hold together had to track that itself and hope the machine stayed up, which is why banks reconciled their books overnight and fixed the discrepancies in the morning.

The fix is not the one people assume. You cannot make two writes happen simultaneously, so databases stopped trying and did something else: they write down what they are about to do before doing any of it.

That record goes into a log, a file the database only ever appends to. It is flushed to disk, meaning the operating system is forced to put the bytes on the disk itself rather than hold them in memory where a power cut would erase them. Only then does the database touch the actual rows.

For the fifty dollar transfer, the log holds roughly this:

txn 41  begin
txn 41  account A   100 -> 50
txn 41  account B    20 -> 70
txn 41  commit

The last line decides everything. On restart the database reads the log. If the commit line is there, it applies every change above it. If the commit line is missing, it undoes them. No judgment is involved. One line either reached the disk or it did not.

That is write-ahead logging. Jim Gray described it in Notes on Data Base Operating Systems in 1978, and in 1983 Härder and Reuter named the guarantees it provides: ACID, whose A is atomicity.

Atomicity is not what the word suggests. The two writes never happen at the same instant. They happen one after another, and the machine can die between them. What makes them atomic is that one commit line. It was written before either change was applied, and it decides afterwards whether both count or neither does.

That only works because the database owns both accounts. They are rows in its own files, covered by its own log, so a single commit line can settle both at once.

A charge at a payment provider is not a row in your database. Your log cannot describe it, your commit line cannot settle it, your rollback cannot undo it. Nothing marks that edge in your code either: no error, no type mismatch, the two calls look the same.

So you need the 1978 trick without the database doing it for you. Somewhere you control, write down "I am about to charge this card" before you charge it, so a crash leaves a note behind and the next process can work out what happened.

1978 to 1991: trying to leave the box

One database's log can promise nothing about what happened in another, and the obvious move is a bigger transaction spanning both.

That runs into an obstacle nobody has removed since. Akkoyunlu, Ekanadham and Huber published the impossibility proof in 1975, and Gray named it the Two Generals paradox in that same 1978 paper: you send a request, it times out, and you cannot distinguish "it never arrived" from "it arrived, ran, and the acknowledgment was lost coming back." Those two worlds look identical from where you stand, and they still look identical after you add retries, better timeouts or a service mesh. The ambiguity is not a gap in your tooling.

The field built two-phase commit anyway. A coordinator asks every participant to promise it can commit, waits for all of them to agree, then tells everyone to proceed. Standardized as X/Open XA in 1991, it delivers genuine atomicity across systems.

The cost is availability. To keep its promise, each participant has to lock the rows involved, meaning hold them so that nobody else can read or change them until the outcome is settled. If the coordinator dies between the promise and the instruction to commit, every participant sits holding those locks and waiting for orders that are not coming. It also requires every participant to implement the protocol, which looks harmless in 1991 and proves fatal twenty years later.

Two-phase commit also assumes the work is short enough to hold locks through. What if it isn't?

1987: work that takes too long

Book a trip, meaning a flight, then a hotel, then a car. Wrap all three in one transaction and you hold locks for the entire booking while a human decides whether they want the aisle seat, and everything touching those rows queues behind you. Garcia-Molina and Salem wrote this up at SIGMOD in 1987: long-lived transactions hold resources for long periods and badly delay the shorter, more common transactions around them.

Their answer was the saga, which replaces one long transaction with a sequence of short ones that each commit immediately, pairing every step with a compensating step that undoes it. If step four fails, you walk backwards through the compensators for three, two and one.

Compensation is not rollback. You cannot recall a confirmation email or erase a charge the way a database undoes a write, so you cancel the booking, issue the refund, send the correction. The undo happens in meaning rather than in bytes, which makes it visible to everyone and sometimes involves apologizing to a real person.

That changes how you order steps. Reversible work goes before the step past which you cannot turn back, which goes before work that can only move forward. Get it wrong and you will need to compensate something that has no compensator.

The cost is isolation, which a saga gives up by construction. Isolation is the promise that nobody sees your work until it is finished, and a saga cannot make that promise because its steps commit one at a time. Other processes will observe your partially completed work, and you decide in advance whether that is acceptable for each step.

Remember the phrase long-lived transaction. A model call taking thirty seconds is one, and a human approval taking three days is an extreme version.

Sagas still assume the other side cooperates. What happens when it won't?

2007 to 2019: giving up gracefully

Two-phase commit needed every participant to speak the protocol, and then the internet arrived and almost nothing did. Neither Stripe nor S3 nor whatever external API you depend on implements XA, and none of them ever will, so for most real operations the bigger transaction is not slow or awkward but unavailable.

Pat Helland had spent much of his career advocating for exactly these guarantees before publishing Life Beyond Distributed Transactions: an Apostate's Opinion in 2007. The title is not ironic. People building large systems had already stopped assuming distributed transactions regardless of the textbooks, and his argument was that the field should work out what to do instead.

What they worked out is a chain of three answers, each one forced by the answer before it.

One. Since you can never know whether your call happened, stop trying to know and make it safe to run twice. That property has a name, idempotence: running something twice leaves the world in the same state as running it once, which is not the same as getting the same answer back twice.

Recognizing a repeat requires a stable identity on every action. That identity is an idempotency key, and Helland wrote the definitive treatment in 2012 as Idempotence Is Not a Medical Condition. With it you stop chasing exactly-once delivery, which is not achievable, and settle for at-least-once delivery plus idempotent processing, which behaves like once.

Two. That handles the external call but not the agreement between it and your own database. Since the two cannot commit atomically, fall back on 1978 and write the intent down first: in one transaction in your own database, the business row and a row saying "I intend to charge this card, under this key." Either both land or neither does, because both sit inside the same guarantee. A separate worker then reads intent rows and performs the calls, retrying until confirmed.

This is the transactional outbox, and the failure it fixes finally gets its name, the dual write problem: two writes to two systems, no coordination, and a window in the middle where a crash leaves you inconsistent. That window is the one in your agent loop. The outbox itself is not new, it is write-ahead logging moved one layer out of the database and into your application.

Three. The last piece arrived when people tired of writing saga logs and outbox relays by hand, badly. Durable execution engines record every completed step to a log that is only ever appended to, then replay it after a crash to rebuild where the process had reached. The lineage runs from Amazon SWF through Cadence, built by Maxim Fateev and team at Uber in 2015 and open sourced in 2017, to Temporal in 2019. Replay only reconstructs the past if running your code again makes the same decisions, which is why determinism stops being a matter of taste in these systems and becomes structural.

That is the inheritance: identity, intent written first, semantic undo, recorded steps. So how much of it did agent frameworks actually pick up?

Where agents sit

Two things write to your database on every turn: the framework's checkpointer, saving loop state, and your own tool code, writing the orders and the charges. In many production systems they point at the same Postgres instance and are never in the same transaction, which makes them the dual write problem arriving as a framework default rather than as anyone's decision.

The consequences are sitting in public issue trackers, three of them describing the same seam:

LangGraph #9006 asks what the intended contract even is when a worker dies in the middle of a tool call. The reporter's framing is the sharpest summary I have read: many traces filed as "the agent failed" are really this seam, where the model never chose badly and the harness duplicated the work.
Strands #4338 reaches it from a different angle. A hook raises after the tool completes, the result message never gets appended, and the conversation is left ending in an unresolved tool call that the agent replays. The issue names payments and order creation explicitly.
Microsoft Agent Framework #7458 executes an approved tool before the run can fail, leaving the retry indistinguishable from a request that was never pending.

So far this is rediscovery, and the microservices answer applies unchanged. Agents add two wrinkles that are in none of the older papers.

The first is that the idempotency key cannot come from the model. Deduplication only works if a repeat is recognizable, and a model that plans again will express the same intention with different wording, different argument order, sometimes a different tool. The second attempt is semantically identical and syntactically new, so a check built on the request body sails past it and the card is charged again. The key has to come from somewhere stable and outside the model's control, which in practice means position in the plan, something like run_id:step_index.

The protocol has not caught up either. MCP, the Model Context Protocol that most agent tooling now speaks, has no idempotency key field, and JSON-RPC request IDs are regenerated on every retry, so a retried tool call is indistinguishable from a new one at the wire level. SEP-3182 proposed fixing that with an idempotencyKey alongside arguments on tools/call, rejecting any key reused with different arguments. It has been closed.

The second is that the determinism requirement is violated by design. Durable execution rebuilds state by replaying your code, assuming it makes the same choices twice, and you have installed the least deterministic component ever shipped at the center of the loop. The way out is to treat the model call as a result to look up rather than logic to execute again, recording the completion when it first happens and reading it back on replay.

Four separate papers in 2026 independently rebuilt transactional machinery for agents, which is decent evidence the gap is real. It is also worth noticing that Cordon's effect outbox is the transactional outbox and SagaLLM's compensation is the 1987 saga, so the field is working this out from scratch rather than reading it.

None of which you need in full to stop the double charge. What is the smallest version?

The smallest thing that fixes it

One table and one rule. The rule is from 1978: write down what you are about to do, before you do it.

create table agent_actions (
  key        text primary key,
  tool       text        not null,
  args       jsonb       not null,
  status     text        not null default 'pending',
  result     jsonb,
  created_at timestamptz not null default now()
);

Every tool call now begins by looking up its key, and there are only three things it can find.

What you find	What it means	What you do
No row	First attempt	Write the row, then call the tool
Row marked `done`	It already happened	Return the stored result, do not call
Row marked `pending`	An earlier attempt died mid-call, and you cannot know whether it landed	Call again, with the same key

That is the entire design. The code is that table:

def run_tool(conn, run_id, step, tool, args, tools):
    key = f"{run_id}:{step}"          # identity from position, never from the model

    with conn.transaction():
        claimed = conn.execute(
            "insert into agent_actions (key, tool, args) values (%s, %s, %s) "
            "on conflict (key) do nothing returning key",
            (key, tool, Json(args)),
        ).fetchone()

    if claimed is None:                                  # a row already exists
        prior = conn.execute(
            "select tool, args, status, result from agent_actions where key = %s", (key,)
        ).fetchone()

        if (prior.tool, prior.args) != (tool, args):
            raise PlanDivergence(key, prior.tool, tool)  # see below

        if prior.status == "done":
            return prior.result                          # row two: already happened

        # row three: still pending, so fall through and call again, keyed

    result = tools[tool](**args, idempotency_key=key)
    conn.execute(
        "update agent_actions set status = 'done', result = %s where key = %s",
        (Json(result), key),
    )
    return result

Why this works: the row commits before the tool is called. A crash after that point always leaves a note behind, so the resumed run finds the row instead of a blank slate and never gets to decide freshly about something already done.

Row three is the honest one. It does not pretend to know whether the charge landed, it hands the same key to the payment provider and lets their deduplication settle it. That is the at-least-once plus idempotency bargain from the last section, in one branch.

There is a fourth thing you can find, and it is the one that catches people out. The row exists, but for a different action, because a key built from position assumes step three means the same thing on every attempt and a replanning model can put something else in that slot. Without the check you would hand an old result to a new intention. Refuse instead. MCP's own idempotency proposal reached the same conclusion when it specified that a key reused with different arguments is rejected outright.

What each additional piece buys

So what does that leave unsolved? Three things, and you will hit them in this order.

One. A row can sit in pending forever if the process that claimed it never returns, so you add a worker that scans for stale rows and finishes them. That worker is the outbox relay, and it also lets your agent step commit the intent and return immediately rather than blocking on a slow external call.

Two. A plan with several steps can still end up half applied, because step three succeeding does not help when step four fails permanently. So each reversible step gets a compensator, the irreversible ones move to the end, and you walk backwards on failure. That is the 1987 saga, and you need it only once an action spans more than one external system.

Three. A long run can lose expensive model calls on every crash, and asking the model again means the replayed agent may take a different path than the one you have already partly executed. So you record model completions alongside tool results. That is durable execution, and by this point adopting Temporal or DBOS is usually cheaper than continuing to build it.

Then break it deliberately, because almost nobody does and it is the only way to know any of this works.

Kill the process after the charge but before the status update. The next attempt should find a pending row and lean on the key rather than charging blindly.
Kill it between the business commit and the checkpoint write, which is the bug from the top of this article. The resumed run should find the action row and skip the call.
Kill it halfway through a plan with several steps. The compensators should run in reverse order.

Each is about five lines of test, and each corresponds to an outage you would otherwise have later.

Memories, guesses and apologies

All of that is machinery. It is worth asking what the machinery is for, and the best answer I know predates every framework in your stack.

In 2009, Helland and David Campbell wrote Building on Quicksand, arguing that once you accept that no component has complete knowledge, the only honest model left is memories, guesses and apologies. You remember what you have seen, you act on incomplete information and are sometimes wrong, and you build a mechanism for making it right afterwards.

That maps onto an agent almost too neatly: the model is the guess machine, the message list is the memory, and the compensating transaction is the apology, which is the only one of the three most agent stacks have not built. We assembled the guess machine, bolted the memory onto the side, and the apology is still missing from most production systems.

The work of figuring out what that apology should look like was done decades ago by people already burned by the same class of failure, and the papers are all still sitting there.