Idempotency keys for AI agents: why hashing the arguments doesn't work

After our last post on why our agents refuse to redo work after a crash, a reader left a comment that took the discussion a level deeper.

The point, roughly: you can't cleanly resume a non-deterministic trajectory, so the only thing that really holds is idempotency at the effect boundary. And then the question that stuck with me: why make the orchestrator decide what's safe to replay, instead of pushing that decision down to the tools themselves?

They're right, and this post is my attempt to answer properly. It covers what we do today, why the obvious version of that idea breaks with LLMs, and the design we're converging on. Fair warning: the last part isn't shipped. It's a proposal, and I'd genuinely like it torn apart.

TL;DR

Today, ApowerB refuses to auto-replay any run that called any tool, reads included. Safe, blunt, and stuck the moment a run completes halfway.
The fix is per-tool idempotency keys, so a replay dedupes at the tool. But hashing the arguments doesn't work when an LLM writes them: the replay drafts a slightly different email, and the hash changes.
The rule that holds: build the key from stable intent, never from generated content.
We already do this in two places, and enforce "read-only" by construction in a few others. The rest is a design sketch, and an open invitation.

Where we are today: default-deny

When a run fails, our runtime records which tools it called. If it called anything, an automatic replay is refused with a 409 that lists the tools (run_main.py). A human can still force it.

We don't classify tools. A GET counts the same as a send_email. That sounds lazy, so let me defend it for a second, because the reader's own examples are the best argument for it:

a GET against a rate-limited or stateful API is not free to repeat;
a "pure" lookup that writes an audit row isn't pure;
a tool from a third-party MCP server can call itself read-only, and you have no way to check.

Inferring purity gets exactly these cases wrong, and they're the ones that hurt. Refusing everything costs a click. Guessing wrong costs a duplicate email to a customer.

Where default-deny breaks

Default-deny is safe. It is also stuck the moment a run finishes halfway.

Picture an email campaign. Our email-campaign skill already puts a human gate in front of the effect: it drafts everything, then never sends without explicit approval. Once approved, it loops over the recipients and sends one email each.

Now the pod dies after email 40 of 100.

Replay is refused, because tools were called. Correct, and useless: 60 people never get their email.
Force the replay, and 40 people get it twice.

There is no good answer at the run level. The run is the wrong unit. The email is the unit. That's exactly where the reader's idea comes in: let each effect know whether it already happened.

The obvious version, and why it breaks

The textbook approach: every effectful tool computes an idempotency key from its arguments, say sha256(json(args)), and the runtime skips any call whose key it has already executed.

With a normal program, that works. With an LLM it quietly fails, because the arguments aren't stable either.

First attempt, the model calls:

{"to": "anna@acme.com", "subject": "Following up on your request",
 "body": "Hi Anna, following up on the quote you asked for..."}

Replay, same input, same agent:

{"to": "anna@acme.com", "subject": "Your quote",
 "body": "Hello Anna, I'm getting back to you about your quote..."}

Different subject, different body, different hash. The ledger has never seen this key. Anna gets two emails, and your idempotency layer reports that everything went perfectly.

The reader's line from the last thread applies one level down: step 3 isn't a stable coordinate. Neither are the arguments.

The rule: stable intent, never generated content

Split every argument into three kinds:

Kind	Examples	In the key?
Identity from the outside world	event id, message id, thread id, record id, normalized recipient address	Yes
Derived deterministically from inputs	hash of an uploaded file's content	Yes
Generated by the model	body, subject, summary, any free text	Never

The key answers "have I already done this thing, for this reason, to this target?", not "have I already sent these exact bytes?". For the campaign: key = (triggering_event, tool, recipient). The replay can rewrite every email from scratch; Anna's key is the same, so she's skipped.

Two places where we already do this

This isn't hypothetical in our codebase. It's just not generalized yet.

1. Webhooks: the key is the provider's identity. When Microsoft Graph re-delivers a notification, a unique index on (subscription_id, resource_id) rejects the duplicate insert, and the message is processed once (outlook.py). Nothing generated goes into that key. It's pure outside-world identity.

2. BI datasets: the key is derived from the content. When an agent imports a file as a dataset, the dataset's ID isn't random. It's a UUIDv5 over who owns it, where it lives, and a SHA-256 of the bytes (bi_datasets.py):

digest = hashlib.sha256(attachment["content"]).hexdigest()
dataset_id = str(uuid.uuid5(
    uuid.NAMESPACE_URL,
    "|".join([owner.lower(), organization_id, project_id,
              attachment["folder"], attachment["name"],
              data.sheet or "", digest]),
))
existing = await store.get(dataset_id)
if existing is not None:
    return {"success": True, "dataset_id": dataset_id, "reused": True}

Run it twice on the same file: one dataset. Change one byte: a new one. Notice what's not in the key: the display name, which is formatted text.

And here's the scar that comes free with it. What if the user deleted that dataset? Soft deletes keep the row, so the deterministic ID is still taken, and the insert collides. Our code catches the IntegrityError and falls back to a fresh random ID. That's the general lesson: an idempotency key outlives the thing it points to. Your ledger needs to know the difference between "already done" and "done, then undone".

And some tools are pure by construction, not by declaration

The other half of the reader's question was classification. Our answer so far: we don't trust a tool that says it's read-only. We trust one that can't write.

The Oracle tool opens a SET TRANSACTION READ ONLY and rolls back (oracle.py).
The MongoDB tool refuses aggregation pipelines with a write stage ($out, $merge) (mongodb.py).
The Redshift tool validates that the SQL is read-only before it runs (redshift.py).

These are the only tools we'd be comfortable marking "safe to replay", and only because the guarantee lives in the tool's code, not in a flag next to it.

The design we're converging on

This part is a sketch, not shipped code. It's the shape we think is right, written down so it can be challenged.

1. Tools declare their effect, and default to the worst case.

@tool(effect="none")            # pure by construction, enforced in the tool
def oracle_run_query(sql: str): ...

@tool(effect="idempotent",
      key=lambda args, ctx: (ctx.trigger_event_id, "send_email",
                             normalize(args["to"])))
def send_email(to: str, subject: str, body: str): ...

@tool                            # no declaration: treated as unsafe
def third_party_mcp_action(**kwargs): ...

2. An effects ledger, with a unique constraint. Before an idempotent tool runs, the runtime looks its key up in a ledger table. Already there: return the stored result, don't execute. Not there: execute, then record. The same unique-index trick as our webhooks, applied to every effect.

3. Undeclared stays default-deny. Anything without a declaration, including every third-party tool, keeps today's behavior: if it ran, the run isn't auto-replayed. Declaring a tool idempotent is how you earn automatic replay, one tool at a time.

4. Pass the key through when the provider supports it. Some APIs accept an idempotency key natively (Stripe's Idempotency-Key header is the classic example). When they do, the key should travel all the way to the provider, so even a crash between the API call and our ledger write dedupes on their side.

With that in place, the campaign that died at email 40 replays cleanly: 40 ledger hits, 60 sends, nobody gets two emails.

The questions I don't have good answers to yet

This is where I'd genuinely like help:

Which fields go in the key, per tool? For email, (event, recipient) feels right. For "post a summary to Slack", is it (event, channel)? What about an agent that legitimately sends two different emails to the same person for the same event?
What if the replay takes a different branch? The model may decide to email a different set of people the second time. Per-target keys handle overlap, but what about the people who were emailed in run one and dropped in run two?
How long does the ledger remember? Forever is wrong (storage, and "the same email next month" is legitimate). Too short reopens the duplicate window.
The crash between the effect and the ledger write. If the provider has no native idempotency, there's a gap. Is "record intent before, confirm after" plus reconciliation good enough?

Come build it with us

If you've shipped something like this, or you have strong opinions about any of the questions above, we'd love your help. We're opening an RFC issue on tool-level idempotency keys, and every contributor is credited by name.