Your HTTP client raises ETIMEDOUT on POST /charges. Something has to be written to the database, the status column has two useful values, and so the catch block writes failed.
The timeout isn't the bug. Timeouts are normal and you'll get another one this week. The bug is that failed is a claim about the outside world, and at the moment you write it you are not in a position to make any claim at all. The provider may have taken the money 200 ms ago. You have asserted, in durable storage, that it didn't.
Everything downstream now believes you. The customer gets an error and pays again. Support issues a refund for a charge your system says never happened. Reconciliation, six hours later, finds a movement on their side with nothing on yours and files it as a break, which is the only part of the system that behaved correctly.
Three outcomes, one symptom
A request that doesn't come back has three possible histories, and they are indistinguishable from where you're standing:
| What happened | What you saw |
|---|---|
| The request never reached them | Timeout |
| It reached them, they committed, the response was lost | Timeout |
| It reached them, they committed, your process died before writing | Timeout |
There's a fourth that's worth separating out, because it breaks the mental model people fall back on. The provider accepted and committed, and the underlying rail rejects it forty minutes later. That one isn't ambiguous, it's just slow, and it means "confirmed" is itself a state with a subsequent transition rather than an end.
The standard error handler collapses all of this into a boolean. try succeeded, catch failed. That shape is fine when the failure costs you a retry and nothing else. It is wrong the moment the operation moves money, because in this domain the third outcome is not an exception case. During the incidents where processes are dying and connections are being reset, which is exactly when you most need to be right, it's the common case.
The correction is small to state and awkward to retrofit: unknown is a state your system holds, not an error it handles.
provider_ref IS NULL is not a state machine
Most systems already encode the unknown accidentally. There's a payments row with status = 'failed' and a null provider_ref, and someone on the team knows that a null there sometimes means "we never got a response" and sometimes means "we never made the call". That knowledge lives in one person and expires when they change teams.
The distinction has to be in the enum, because the enum is what queries filter on and dashboards group by.
create type payment_state as enum (
'intended', -- we have decided to pay, nothing has been sent
'in_flight', -- a request is on the wire right now
'unknown', -- the request ended without an answer
'confirmed', -- the provider has told us it happened
'settled', -- it appeared in a statement we ingested
'rejected' -- the provider has told us it did not happen
);
Two things about that list are load-bearing.
unknown and rejected are different states, and only one of them is a fact. rejected means the provider gave you an answer. unknown means nobody has. Merging them is precisely the mistake that started this post.
And confirmed is not the last state. The provider saying yes over an API is their intent, not the money's arrival. Only settled is backed by a statement, which is the part reconciliation exists to establish. A system that stops at confirmed has substituted a promise for the evidence.
Write the intent down before you make the call
The dangerous window isn't the call. It's the gap between deciding to pay and recording that you decided.
// Broken. A crash on line 1 leaves no trace that money may be moving.
const charge = await provider.charge(amount, card);
await db.payments.insert({ providerRef: charge.id, state: "confirmed" });
If the process dies inside provider.charge, there is no row. Nothing to reconcile, nothing to poll, nothing to alert on. The money left and your database has never heard of it. This is worse than a wrong state, because a wrong state at least shows up in a query.
Commit the intent first, then call:
const payment = await db.payments.insert({
state: "intended",
amount,
idempotencyKey: key, // generated here, stored here, reused forever
reference: ourReference, // the join key reconciliation will need
expectedResolutionBy: addBusinessHours(now, 2),
});
await db.payments.setState(payment.id, "in_flight");
try {
const charge = await provider.charge(amount, card, { idempotencyKey: key });
await db.payments.confirm(payment.id, charge.id);
} catch (err) {
await db.payments.setState(payment.id, classify(err));
}
This is write-ahead logging with the business as the log. The row exists before the side effect, so every crash leaves something behind that knows to go and ask.
Note what classify is allowed to return. A response that says no is rejected. A connection that was refused before any bytes left the machine is rejected too, because the request provably never arrived. Everything else, including every read timeout, is unknown. If you can't tell which kind of timeout you got, it's unknown. Guessing in the safe direction here costs you a poll; guessing in the other direction costs you a duplicate payment.
The idempotency key belongs on that row for the same reason. It's a property of the intent, not of the attempt, which is the contract the retry depends on.
Unknown needs a deadline and an owner
A state that nothing is responsible for leaving is just a slower way of losing data. unknown is only useful if something is obliged to resolve it, and the obligation has to be written into the row rather than into a runbook.
That's what expected_resolution_by is for. It gives you two queries that a human can act on:
-- Working as designed: young unknowns, being polled.
select count(*) from payments
where state = 'unknown' and expected_resolution_by > now();
-- The alert. Something is stuck and a person needs to look.
select * from payments
where state = 'unknown' and expected_resolution_by < now()
order by created_at;
Alert on the age of the oldest unknown, not on the count. Volume tracks traffic and provider health, and it's noisy in a way that trains people to ignore it. Age tracks whether resolution is working at all. Fifty unknowns that all clear within four minutes is a healthy system on a bad network. One unknown from Tuesday is an outstanding question about real money, and it will still be there in March unless someone is told.
The pressure you have to resist is organisational, not technical. A queue of unknowns is uncomfortable to look at, and somebody will propose a nightly job that ages them out to failed so the dashboard goes green. That job converts an open question into a false answer, which is the only genuinely irreversible operation in this whole design.
Ask, but know which answer is authoritative
There are three ways an unknown resolves, and they differ in latency and in how much you should believe them.
Query the provider by your own key. Fastest, and available on demand. Most payment APIs let you look up by idempotency key or by your reference; the ones that only let you look up by their ID are useless here, because their ID is exactly what you didn't receive. This is the check you run on a backoff schedule.
Wait for a webhook. Free and quick when it works, and it is not a plan on its own. Webhooks are at-least-once at best, they arrive out of order, and they get dropped by your own load balancer during the incident that caused the unknowns in the first place. Treat an inbound webhook as a hint that makes you poll, not as the source of truth.
The webhook race is worth naming, because it produces a bug that looks impossible. Their event can arrive before your own transaction commits. Your handler looks up the payment, finds nothing, logs "unknown webhook" and drops it. The confirmation you needed most is the one you threw away. Persist unmatched webhooks in their own table, keyed by the reference they carry, and re-match them when the payment appears.
Read the settlement file. Slowest, usually a day, and the only one of the three that constitutes evidence. The API says what the provider believes. The statement says what the bank did. When those two disagree, and they do, the statement wins.
So the polling loop is a convenience that closes most unknowns in minutes, and the file is the backstop that closes the rest. Build both. A system with only the poll has no answer when the provider's API is the thing that's broken.
Money in flight is a balance
Here's the part that turns this from an engineering pattern into something the finance team can use: an unknown isn't only a row state, it's an amount, and that amount has to be somewhere in the ledger.
The usual instinct is to write no entries until the payment confirms. That keeps the ledger clean and makes it lie: the customer's money has left their account and, according to your books, has not gone anywhere. Double entry has an answer for this that predates all of us, which is a suspense account.
On intent: DR Customer 100.00
CR Payments in transit 100.00
On confirmation: DR Payments in transit 100.00
CR Provider clearing 100.00
On rejection: DR Payments in transit 100.00
CR Customer 100.00
Every leg is a new entry rather than an edit, so the history of an unknown that eventually resolved is still readable a year later. That's the same append-only discipline reconciliation depends on.
What you get for it is a number: the balance of "payments in transit" is your total exposure to unresolved outcomes, in currency, at any instant. Graph it. It should be small and it should spike and drain. A balance that only grows means resolution has quietly stopped working, and you'll see that on a chart weeks before anyone opens the payments table.
It also gives the unknown a home in the accounts. Nobody has to invent a policy for it during a close, because it was already a line item.
Cancelling an unknown is another unknown
The tempting shortcut, when you have a payment you're unsure about, is to void it and start clean.
You can't. POST /charges/void needs the charge ID you never received, and the variants keyed on your reference have the identical failure mode: their response can time out too. Now you have an unknown cancellation of an unknown payment, and the state space has squared rather than shrunk.
Unknowns resolve by asking, not by acting. The only safe operations while you're in that state are reads. Once you know a charge exists, refunding it is a normal, well-defined thing with its own record, and a refund is a new fact rather than an erasure of the old one.
This is also why the answer to "should we retry?" is no, not until you've asked. Retrying a read timeout is safe if and only if the idempotency key travels with it and the provider still recognises it. Both halves have to hold. Their key window is often shorter than the age of your oldest unknown, and once it lapses, your retry is a brand new charge as far as they're concerned.
Your timeout is a policy, not a network fact
The number of unknowns you have to handle is partly your own choice, made in a config file, usually by accident.
If your client's read timeout is 10 seconds and the provider's p99 is 14, you are manufacturing unknowns at roughly one percent of traffic, deterministically, forever. Every one of those requests is going to succeed on their side. You just decided not to wait for the news. Look up the number, don't assume it: the ones the provider publishes are for the happy path, and card authorisations that route through 3-D Secure or a slow issuer are not the happy path.
Two adjustments follow. Set the read timeout above the provider's real tail latency, and give the write path its own timeout budget rather than inheriting the API gateway's. And separate the two timeouts in your client configuration, because connect and read mean genuinely different things here: a connection you never established is a request that never happened, and a response you never read is a question.
Then check what your infrastructure does underneath you. A 30 second idle timeout on a load balancer, a proxy that retries idempotent methods on its own, a client library that quietly retries once by default: any of those turns a clean request into an ambiguous one without appearing in your code. The library default is the one that catches people, because it's a retry you didn't write and therefore didn't attach a key to.
What the customer sees
None of this reaches the user as "unknown". They get three states, and the middle one is the whole point of the exercise:
- Processing. Honest, and the correct answer for as long as you're unsure. Give it an expected duration so it doesn't read as a hang.
- Confirmed, once you have an answer you'd defend in a dispute.
- Failed, only when the provider said no.
The thing you have to stop is the customer resolving the ambiguity for you by pressing Pay again. Their second attempt is a genuinely different request with a different key, and idempotency will process it exactly as instructed. What stops it is a business uniqueness constraint: one non-terminal payment per invoice, enforced in the database, not in the button's disabled attribute.
create unique index one_open_payment_per_invoice
on payments (invoice_id)
where state in ('intended', 'in_flight', 'unknown');
That partial index is about fifteen minutes of work and it removes the most common way an unknown becomes a duplicate charge: an anxious human with a working mouse.
What this doesn't buy you
- Not fewer unknowns. The count is set by network reality and your timeout policy. This makes them visible, bounded and resolvable, which is the whole available win.
- Not idempotency. Modelling the state doesn't make the retry safe. The key does, and it has to have been on the row before the first attempt.
- Not reconciliation. Polling closes an unknown against the provider's opinion. Only the statement closes it against the bank's.
- Not correctness. A payment can be fully confirmed, fully settled, perfectly reconciled and still the wrong amount to the wrong person. Nothing here has an opinion about intent.
- Not a resolution guarantee. Some unknowns need a human and a phone call. The design's job is to make sure there are three of those a quarter and that each one is found in hours, not that the number reaches zero.
The short version
- A timeout is not a failure. Writing
failedon one is a claim you can't support. -
unknownbelongs in the enum, next torejectedand distinct from it. - Commit the intent, with its key and its reference, before the call that can vanish.
- Only a connection that was never established counts as "never happened". Every read timeout is unknown.
- Give every unknown a resolution deadline, and alert on the age of the oldest, never the count.
- Poll by your own key. Treat webhooks as hints, and store the ones that arrive before the payment exists.
- The settlement file is the only authority. The API is an opinion, however confident.
- Put money in flight in a suspense account, so the exposure is a balance somebody can graph.
- Never cancel an unknown. Reads only, until you know.
- One open payment per invoice, as a partial unique index, or the customer will resolve the ambiguity for you.
Two things to go and look at today. Grep your codebase for catch blocks around a provider call and read what each one writes to the database; every status = 'failed' on a timeout is a future refund request for a charge you'll swear never happened. Then find your client's read timeout and put it next to the provider's p99 latency. If the first number is smaller, you already know how many unknowns you're creating a day, and you've never seen a single one of them.
Originally published at mashhadi.me.
