Custodial ETH withdrawals: the nonce is a database row, not a node call

A custodial wallet service has to turn a user's withdrawal request into exactly one on-chain transaction, while every layer around it — HTTP clients, queues, restarts, impatient users — is at-least-once. The first thing that breaks is the nonce. Two workers ask the node for the next transaction count, both get the same number, both sign, and one transaction quietly replaces the other on the network. Nothing throws. The user sees a confirmation and no coins.

I have built custodial wallet services for BTC and ETH and the on-chain payment system around them. Account-model chains punish sloppiness here more than UTXO chains do, because the nonce is a strictly sequential resource owned by the sending address, and nothing in the JSON-RPC API will defend it for you.

The node's nonce is a report, not a reservation

eth_getTransactionCount answers one of two different questions depending on the block tag: how many transactions from this address are mined (latest), or how many the node currently holds in its own mempool view (pending). Neither one is an allocation.

latest ignores everything in flight, so under any concurrency it hands the same number to everybody. pending reflects a single node's mempool, which is not a consensus object: it evicts transactions under memory pressure, it cannot see what a sibling node accepted, and it forgets everything on restart. Building a withdrawal queue on pending means your correctness depends on a cache you do not own and cannot audit after the fact.

So the rule I end up at every time: the nonce is allocated by my database, in the same transaction that persists the signed payload. The node is told about the result afterwards, as many times as necessary.

Three states, not one

"Send a withdrawal" is not one operation. It is three, and they have completely different repeat semantics.

Most broken pipelines I have inherited collapse signed and broadcast into one sent state. After a crash they cannot distinguish "we never built this transaction" from "we built it and the RPC call timed out" — and those two have opposite recovery paths. One should rebuild, the other must never rebuild.

Allocating the nonce

create table hot_account (
    address    text   primary key,
    next_nonce bigint not null
);

create table withdrawal (
    id           bigserial primary key,
    user_id      bigint        not null,
    from_address text          not null references hot_account(address),
    to_address   text          not null,
    amount_wei   numeric(78,0) not null,
    state        text          not null default 'requested',
    nonce        bigint,
    raw_tx       bytea,
    tx_hash      text,
    created_at   timestamptz   not null default now()
);

create unique index withdrawal_nonce_uniq
    on withdrawal (from_address, nonce)
    where nonce is not null;

Two details there matter more than they look.

The partial unique index is the invariant, expressed in the only place that can actually enforce it under concurrency. If a bug ever lets two workers allocate the same nonce, I want a constraint violation and one stuck withdrawal, not two valid signatures racing on the network.

And next_nonce lives in its own tiny table so it can be locked on its own. That lock serialises every withdrawal leaving that address, so the critical section has to stay short — which is exactly why no network call belongs inside it.

from eth_account import Account

GAS_LIMIT = 21_000  # plain value transfer

def allocate_and_sign(conn, withdrawal_id, key, chain_id, max_fee, priority_fee):
    with conn:  # one database transaction
        cur = conn.cursor()
        cur.execute(
            'select state, from_address, to_address, amount_wei '
            'from withdrawal where id = %s for update',
            (withdrawal_id,),
        )
        state, from_address, to_address, amount_wei = cur.fetchone()
        if state != 'requested':
            return  # already allocated by someone else; not our job

        cur.execute(
            'select next_nonce from hot_account where address = %s for update',
            (from_address,),
        )
        (nonce,) = cur.fetchone()

        signed = Account.sign_transaction(
            {
                'chainId': chain_id,
                'nonce': nonce,
                'to': to_address,
                'value': int(amount_wei),
                'gas': GAS_LIMIT,
                'maxFeePerGas': max_fee,
                'maxPriorityFeePerGas': priority_fee,
            },
            key,
        )

        cur.execute(
            'update hot_account set next_nonce = next_nonce + 1 where address = %s',
            (from_address,),
        )
        cur.execute(
            "update withdrawal set state = 'signed', nonce = %s, raw_tx = %s, tx_hash = %s "
            'where id = %s',
            (nonce, signed.raw_transaction, signed.hash.hex(), withdrawal_id),
        )

The ordering is the whole point. The lock, the increment, the signature and the stored payload commit together. If the process dies halfway, the transaction rolls back, the nonce is not consumed, and nothing was ever broadcast — the request is still cleanly requested. If it commits, exactly one signed payload exists for that nonce and it is durable before a single packet leaves the process.

Note also what is absent: any RPC call. Signing is local. Fee parameters are read before the transaction opens and passed in; a plain transfer's gas limit is a constant. Fetching fees inside the critical section would put a network timeout inside a lock that every other withdrawal from that address is waiting on.

Rebroadcast is free; re-signing is not

ALREADY_HAVE_IT = ('already known', 'known transaction', 'transaction already exists')

def rpc_message(exc):
    arg = exc.args[0] if exc.args else ''
    text = arg.get('message', '') if isinstance(arg, dict) else str(arg)
    return text.lower()

def broadcast(w3, conn, withdrawal_id):
    cur = conn.cursor()
    cur.execute(
        "select raw_tx from withdrawal where id = %s and state in ('signed', 'broadcast')",
        (withdrawal_id,),
    )
    row = cur.fetchone()
    if row is None:
        return

    try:
        w3.eth.send_raw_transaction(bytes(row[0]))
    except ValueError as exc:
        message = rpc_message(exc)
        if 'nonce too low' in message:
            return  # the slot is already mined; the watcher decides by whom
        if not any(known in message for known in ALREADY_HAVE_IT):
            raise

    with conn:
        cur.execute(
            "update withdrawal set state = 'broadcast' where id = %s and state = 'signed'",
            (withdrawal_id,),
        )

The error handling is the interesting half. already known is not a failure of this withdrawal, it is evidence the network already has the exact bytes I am holding. Sending the same signed payload a thousand times produces one transaction, because the payload is its own identity.

The dangerous instinct is the opposite one: a send times out, and somebody "retries" by rebuilding the transaction with a fresh nonce. Now two signed payloads exist for one intent, and if the first one was in fact accepted, both can mine, and the user is paid twice from a hot wallet. Once a payload is signed, the retry is always the same bytes.

Fee bumps create siblings

Under a rising fee market a transaction can sit unmined for a long time, and because nonces are sequential, everything behind it waits. The fix is a replacement: same nonce, higher fee, new signature. Most clients require a meaningful increase before they will evict the old one — geth's default price bump is ten percent — so the bump has to be real, not cosmetic.

That produces the part nobody warns you about. One withdrawal now has several valid transaction hashes, and only one of them will ever be mined. If the confirmation watcher polls a single stored tx_hash, a successful bump is indistinguishable from a lost transaction, and the operator on shift starts investigating a payment that actually went through.

So attempts get their own table, and the watcher keys on the pair that is genuinely unique on-chain — (from_address, nonce) — rather than on any one hash.

create table withdrawal_attempt (
    id            bigserial primary key,
    withdrawal_id bigint      not null references withdrawal(id),
    tx_hash       text        not null unique,
    raw_tx        bytea       not null,
    max_fee_wei   numeric(78,0) not null,
    created_at    timestamptz not null default now()
);

def settle(w3, conn, withdrawal_id):
    cur = conn.cursor()
    cur.execute('select from_address, nonce from withdrawal where id = %s', (withdrawal_id,))
    from_address, nonce = cur.fetchone()

    mined = w3.eth.get_transaction_count(from_address, 'latest')
    if mined <= nonce:
        return  # the slot is still open; keep rebroadcasting or bump

    cur.execute(
        'select tx_hash from withdrawal_attempt where withdrawal_id = %s', (withdrawal_id,)
    )
    for (tx_hash,) in cur.fetchall():
        try:
            receipt = w3.eth.get_transaction_receipt(tx_hash)
        except Exception:
            continue
        with conn:
            cur.execute(
                "update withdrawal set state = 'settled', tx_hash = %s where id = %s",
                (tx_hash, withdrawal_id),
            )
        return

    raise RuntimeError(f'nonce {nonce} on {from_address} mined by an unknown transaction')

That last line is deliberate. If the slot is consumed and none of my siblings has a receipt, something signed with my key that I did not record. That is not a retry condition, it is an alert.

Head-of-line blocking is the price

Per-address serialisation means one stuck transaction blocks every later withdrawal from that address. This is not a bug to engineer away; it is the chain's model. The only real lever is how many sending addresses you run. Assigning withdrawals to lanes by hashing the withdrawal id gives independent nonce sequences and independent failure domains, at the cost of splitting the hot balance across them and needing a rebalancing job that is itself a withdrawal — and therefore must use the same allocator, not a special path.

What I would do differently

Persist the signed payload before touching the node, always. I have worked on code that broadcast first and recorded the hash after. The gap between those two lines is where a crashed process becomes an untracked transaction spending real funds, and reconciling that afterwards means scanning the chain for your own address to find out what you did.

Never expose a cancel button that deletes the row. Once a nonce is signed and broadcast, the only cancel that exists is a replacement at the same nonce: zero value, self-transfer, higher fee. A cancel that just marks the database row as cancelled leaves a live transaction on the network with a stale user expectation attached to it.

And keep nonce too low visible in metrics even though the broadcaster treats it as benign. In steady state it appears exactly when a slot has just settled. A cluster of it anywhere else is the signature of double allocation, which is the one failure in this design that costs money rather than time.

The nonce is the only part of a withdrawal that cannot be recreated after the fact. Once I started treating it as a row I allocate and a payload I persist — with the node as a broadcast medium rather than a source of truth — withdrawals stopped being the frightening part of a custodial system and became the boring part.

Originally published on polycratia.com — where I write about payment systems, crypto rails and marketplace backends.