Designing a Flash-Sale Seat Reservation System in AWS (Part 3): Holds, Payments, and the Slow Path

TL;DR: After admission, a seat is held for 10 minutes while the candidate pays. The central rule for payments is one-sided: many paths can confirm a booking, but only one can release a seat. Three paths can mark a booking PAID: the browser callback, the gateway's signed webhook, and a background reconciler.

The reconciler asks the gateway for the payment status. Each of the three paths checks with the gateway, and each is idempotent (safe to run more than once). Only the reconciler releases seats. It does so in two cases: the gateway reports that the payment failed or was cancelled, or the hold deadline has passed and the gateway has not reported the payment as complete. If the gateway cannot answer, the hold is extended. A seat is never released when the outcome is uncertain. The "cancel" or "failed" status in the redirect itself is ignored, because anyone can fake it. Durable writes happen outside the request path.

An SQS FIFO queue feeds a worker that writes to DynamoDB with conditional puts (writes that succeed only if a condition holds). The worker never touches the seat counters. DynamoDB TTL and a Lambda driven by DynamoDB Streams are only a safety net for expiry. On launch day, 31.6K holds were created for 20.7K seats, and the queue never held a message older than 4 seconds. The post ends with how we tested the design, in layers and many times over, and what that testing found before launch.

This is the last part of a three-part series. Part 1 covers the architecture and the edge. Part 2 covers how seat admission stays correct.

The hold lifecycle

What states does a booking go through?

A booking has three durable (stored) states:

• HELD: the seat is counted and a payment exists. The record carries a deadline 10 minutes in the future.
• PAID: the gateway has confirmed the payment. The seat is permanent, and a confirmation email is sent.
• RELEASED: the hold ended without payment. The seat is back in the pool, the record is deleted, and the candidate can try again.

If payment creation or the enqueue fails, the booking never reaches HELD. The API undoes the admission before it responds.

The 10-minute hold is a trade-off in both directions. A shorter hold returns unpaid seats sooner, but it punishes anyone who needs some time to approve a payment on their phone. A longer hold keeps seats locked by people who have left and will not pay. Ten minutes was comfortably longer than the gateway's checkout took in testing, and it kept resale fast. On launch day, 31.6K holds were created for 20.7K seats. About a third of the holds expired or failed, and their seats were sold again within the hour.

Challenge 1: payment signals you cannot trust

Why can't the payment redirect decide the outcome?

After paying, the gateway redirects the browser to /payment/callback?paymentID=...&status=success|failure|cancel. The obvious design is to trust that status: mark the booking paid on success, and release it on cancel. That design is broken in both directions:

• An attacker controls the query string. Anyone can type status=cancel for a payment ID they have seen. That would free a seat that is still being paid for, or one that is already paid.
• Redirects can be old or repeated. The browser's Back button, a second tab or a retried request can all deliver the same redirect again, sometimes after the booking has already changed state.
• A redirect may never arrive. The user closes the tab, their phone loses signal, or the redirect times out. The payment completed, but nobody told us.

We considered three designs.

The three confirm paths are:

1. Browser callback. The API accepts the callback only for a payment ID that it issued, and only with the per-payment signature that the gateway returned when the payment was created. It also refuses to charge once the hold deadline has passed. Otherwise we could take money for a seat that the reconciler has already released. It then calls the gateway's execute API under a lock with an owner token (SET NX with a random token, and only the owner of that token can release the lock). So the payment is executed at most once, even if the redirect arrives twice. If the execute call times out, the API queries the payment once, as the gateway's own integration guide requires. It does that only to recover a completed payment.
2. Gateway webhook. The gateway also sends a signed notification (standard AWS SNS HTTP delivery) when a payment completes. The handler verifies the SNS signature and looks up the booking by invoice number. It still queries the gateway and compares the transaction ID before it marks the booking paid. It always answers 200 to a correctly signed message, even when it leaves the decision to the reconciler. That way SNS never builds up a flood of redeliveries.
3. Reconciler query. This path is covered in Challenge 2.

Every confirm path ends in the same conditional update: HELD → PAID, which succeeds only if the record is still HELD with the same payment ID. Whichever path gets there first wins, and the others do nothing. Confirmation emails are sent after that update succeeds, never before.

One race condition shaped the design early. A fast payer can finish before the worker has written the record. The hold is written to DynamoDB asynchronously, so a user who pays within seconds can come back before any HELD record exists. So at admission, the API also stores a copy of the booking record in Redis, keyed by payment ID, with a TTL slightly longer than the hold. If the callback finds no record in the database, it writes the PAID record from that copy.

Challenge 2: exactly one path releases a seat

Who decides that a hold has expired?

The reconciler is a background loop inside the API fleet. A Redis lock with an owner token makes sure that exactly one instance runs it at a time. Every 20 seconds, that instance does a sweep:

1. Clean up stale pending admissions from Part 2: seats whose request died before it reached the database.
2. Find HELD records within 6 minutes of their deadline, or past it.
3. Handle the most overdue records first, 8 at a time, within a 45-second time budget, so the sweep finishes before the next one starts. When many holds expire at the same time, the records that are closest to hurting a user go first.
4. Ask the gateway for the status of each record:
   • Completed: mark it PAID and send the email. This recovers every user whose redirect never arrived.
   • Failed or cancelled, or not completed after the deadline: release the seat, using the payment-guarded undo from Part 2, and delete the record.
   • Still pending before the deadline: leave it alone. The user may still be paying.
   • The query fails three times: extend the hold by 10 minutes, up to three times, then alert a human. We never release a seat only because we could not get an answer, because the payment may have succeeded.

This is the only status query traffic we send to the gateway on the release side. Keeping it in one rate-limited loop keeps our use of the gateway's rate limits predictable.

DynamoDB TTL is not the main timer. TTL (time to live) is set on the record's deadline attribute. But AWS deletes expired items on a best-effort schedule, and the delay can be long. During a sellout, that delay would mean seats that nobody could buy. So the reconciler expires holds itself, on time. TTL, a DynamoDB Stream, and an expiry Lambda form the safety net. The Lambda acts only on deletions made by the TTL service itself (the reconciler's own deletes are ignored), and only if the deleted record was still HELD. It then runs the same payment-guarded undo. If the reconciler already released that seat, the Lambda's undo does nothing.

One bug from QA is worth sharing. Paid records must remove the TTL attribute. One of our paths wrote PAID records with the hold's TTL still set. DynamoDB would have deleted those confirmed bookings 10 minutes later.

Challenge 3: one token, a rate-limited gateway

How do you share an auth token across a fleet?

The gateway's API calls need a short-lived token. The token grant endpoint (the endpoint that issues a new token) is rate-limited to a couple of calls per hour. If every instance fetched its own token, a deploy or a restart would use up the whole allowance in seconds and lock the entire fleet out of payments.

So the token lives in Redis, shared by the whole fleet:

• One refresher at a time. An instance that finds no valid token takes a SET NX lock with a 45-second TTL. Only the lock holder calls the grant endpoint. All other instances wait for the new token to appear, while still respecting their own request deadline.
• Refresh early. The token is cached until a few minutes before it expires, so a request never uses a token at the moment it expires.
• The grant call outlives the request that started it. The grant call runs on its own 30-second deadline, separate from the request that triggered it. A user's 8-second timeout cannot stop a grant halfway and force another one.
• Wait after a failure. A failed grant keeps the lock for 60 seconds instead of releasing it. So the fleet does not retry in a loop and use up the hourly allowance.

Challenge 4: durability off the hot path

How does the database keep up without slowing down bookings?

We considered four ways to store each booking durably.

The details that matter:

• Message group = the first 4 hex characters of the candidate hash. That gives 65,536 groups. The groups spread the load, and ordering within a group keeps one candidate's messages in sequence. High-throughput FIFO mode (DeduplicationScope=messageGroup, FifoThroughputLimit=perMessageGroupId) removes the old per-queue throughput limit.
• Deduplication ID = the candidate hash plus the request timestamp. A retried send is treated as the same message, but a real re-booking is not.
• Visibility timeout of 60 seconds and 5 receives before a message moves to the dead-letter queue. That queue has an alarm that fires on even one message.

The worker on each instance long-polls the queue (it keeps a receive request open and waits for messages instead of asking repeatedly). It writes each record with a conditional put (the write succeeds only if the record does not exist yet). When the condition fails, the worker has to find out why, and a guess is not good enough:

• PAID: the callback got there first. Skip the message.
• HELD with the same payment ID: a repeated delivery of the same message. Delete the message.
• HELD with a different payment ID: a stale record from an earlier attempt that was released, after which the candidate booked again. Replace it, on the condition that the old payment ID is still there. We found this case in QA, and it is why the payment ID is part of the check. Without it, the stale record hid the new payment, and the new booking could never be marked paid.
• The read failed: leave the message unacknowledged and let SQS deliver it again.

The worker never touches the seat counters. Seats belong to Redis and the admission script. Records belong to the worker. When both could decrease a counter, a redelivered message could free a seat that someone still held. Giving each piece of state exactly one owner made that bug impossible.

DynamoDB itself uses on-demand capacity with one table. The partition key is the candidate hash. A GSI (global secondary index) on payment ID serves the callback's lookup. Point-in-time recovery and deletion protection are both on. In the first five minutes the worker wrote about 37,000 write units' worth of records. The table absorbed that without a single throttle, because the queue had already smoothed out the spike.

Two more stream consumers connect to the table, both with failure handling built in:

• The sync Lambda copies every PAID record into a relational database for the back-office app, using upserts (insert or update in one statement).
• Both Lambdas use ReportBatchItemFailures, split a batch in half on error, limit the record age, and send anything they give up on to an SQS on-failure destination. Without this, a dropped stream batch would mean a lost release or a missing paid booking.

The live status document

How do thousands of writers keep one JSON file correct?

The availability counts on the page come from status.json in S3 (see Part 1). The API and the expiry Lambda both write it. Our first version read the file, changed one field and wrote it back. With several writers at the same time, two writes overlapped, and an old write reopened a level that had just filled.

The rule that fixed it: every writer builds the whole document from Redis, where the counters and the open/closed flags live, and overwrites the file. Writers that run at the same time then arrive at the same result instead of competing over a copy. Two latches (one-time flags) keep the writes cheap:

• A 2-second SET NX throttle turns a burst of admissions into a single S3 write.
• The level-full and closed transitions each publish exactly once.

A release clears those latches, so a freed seat reopens the level on the page.

Test, find, fix, repeat

Was this design right the first time?

No. The design in this series is the result of about four months of testing, finding problems, fixing them and testing again. That loop is one of the main reasons launch day was quiet. Every change went through several layers of testing, and each layer answered a different question.

• Unit tests for every package: each branch of the admission script, the gateway client, the webhook signature check and the status document.
• An end-to-end suite on a local copy of the stack, with a mock payment gateway that can be told to fail or cancel. It runs a booking all the way to PAID. It also runs the failure cases: a gateway error, a duplicate booking, a new booking after a released hold, a forged callback, a callback that arrives twice, and a payment that completes before the worker has written the record.
• Full-rate load tests on a stack the same size as production. Each run used 40,000 test candidates, sent 2,000 bookings per second for 25 seconds, and then sent a slower stream of late arrivals and retries. A run passed only if admissions stopped exactly at the cap, DynamoDB reported no throttling, and the counters matched the database afterwards.
• The payment gateway's sandbox, for real payment flows: pay, cancel, let a hold expire, and book again.
• Reviews and audits of the design, the code and the infrastructure. They looked for signals an attacker could fake, rate limits we could use up, and missing permissions.
• Launch rehearsals. We ran the same reset, load-candidates and open steps that we would use on the day, more than a hundred times, from the same pipeline.

Each layer found problems that the others missed. A few examples:

• Load tests: the Redis client's default connection pool was sized for normal traffic, not for a burst. We raised it to 300 connections per instance and ran the test again.
• Sandbox: the counters drifted after a user cancelled and then booked again. The worker found the old record, treated the new booking as a duplicate, and decremented a counter. This is why the worker never touches the counters, and why it classifies every conflict with a consistent read (Challenge 4).
• Testing the undo paths: a rollback that ran twice pushed a level counter below zero. This is why every undo checks and acts in one atomic step (Part 2).
• Reviewing paid records: a paid record could still carry the hold's TTL, so DynamoDB would have deleted a paid booking about ten minutes later. A paid record now has no TTL.
• Security review: the cancel status in the redirect has no signature, so anyone can fake it. Checking each one with the gateway would also use up the gateway's token limit. This is why a redirect can never release a seat (Challenge 1).
• A week before launch: two concurrent writers could leave the status page showing a full level as open. The writers now run in a fixed order, and the page also checks the counts, so a full level can never look open.

None of these problems is unusual. Most of them only show up under a specific timing, order of events or load, and the rest only show up when someone looks for them on purpose. The only reliable way to find them before users do is to test at every layer, many times, and to test again after every fix.

Lessons

What would I tell someone building this next?

1. Decide the one fact everyone competes for in one atomic step, and keep it off the database. Everything else in the system can be eventually consistent if that one decision is exact.
2. Make the request path as short as possible. Admit, create the payment, enqueue, and respond. Every durable write that can happen after the response should happen after the response.
3. Give every piece of state exactly one owner. Redis owns seats, the worker owns records, and the reconciler owns releases. Most of our pre-launch bugs came from two components writing the same state.
4. Treat every external signal as a hint. Redirects, webhooks and query results can be faked, delayed, duplicated or missing. Confirm with the source, make every state change conditional, and let exactly one path take the action that cannot be undone.
5. Never release when you are not sure. If you cannot tell whether someone paid, extend the hold and alert a human. Selling a paid seat twice is worse than holding an unpaid one a little longer.
6. Idempotency must live in the same atomic step as the action. A check followed by a separate action is a race, and under a burst every race is lost sooner or later.
7. Add capacity before a sudden jump in traffic, and load-test with a mock of every external dependency. The mock gateway let us practise the full booking path at full rate as often as we wanted.
8. Plan as much time for testing as for building. Test in layers, from unit tests to full-rate load tests and launch rehearsals. When you find a problem, fix it, add a test for it, and run every layer again. Most of what makes this design correct was learned that way.

The result was the launch day in Part 1: 20,700 seats sold, 99.99% of requests served without a server error, and zero double bookings.