Hopefully you didn't notice, but a couple of weeks ago I sent an issue of my newsletter that never actually got emailed. My dashboard looked fine and it was marked as published, but the email never hit my inbox (or any of yours 😬).
I didn't notice until two days later when I went to look at the weekly statistics and found nothing. Just a Sent status and a bunch of 0's on the screen.
A quick browse through the code didn't turn up anything obvious. Emails are sent via an async handler triggered by an EventBridge event while the status of the newsletter is marked separately in DynamoDB, a classic "db save and domain event publish" pattern. We've all done it, it looks something like this:
await ddb.send(new UpdateItemCommand({ ... }));
await eventBridge.send(new PutEventsCommand({ ... }));
Despite its simple appearance, this is a nasty anti-pattern. Formally known as the dual write problem, if those two actions aren't contained in a single transaction, you're asking for trouble. Why? I'm glad you asked.
Imagine the write succeeds, but the publish doesn't. Data is updated but the follow-on actions that react to the domain event never run (like your email never being sent 🙃). Things fall into a bad state and you (maybe) end up stumbling upon a problem days later.
Ok, so flip them. Put the event first and the data save second. But if the publish succeeds and the db save fails, then you risk acting on stale data, which depending on your domain, is much worse.
Before you ask, no, serverless doesn't make it better. In many cases, serverless makes us too laissez-faire to solve it the "write" way (get it?).
What you're probably doing
Let's say you already knew not to do the data save and the event publish at the same time. I bet I know what you're doing instead. It's a pattern AWS has recommended for years and is all over Serverless Land.
You do your save to DynamoDB, let the change data flow into DynamoDB Streams, process it in Lambda, then fire your domain events from the stream processor.
When it comes down to it, this is legitimately better, and it's the same thought process behind going storage-first. You persist it first, then process it asynchronously. That makes the database write the event trigger, so now your domain events are tightly coupled to the corresponding data change. Plus, DynamoDB Streams retries automatically (and even bisects batches when configured) if there's a processing failure.
Sounds like our dual write problem is solved, right? Wrong. Kind of. It's complicated. 😅
DynamoDB Streams records are change data capture, not domain events. The Lambda function you write to handle them receives a NewImage and OldImage record, and it's up to you to infer what happened upstream. Smells like leaky business logic to me. Before you know it, you have this thousand-line file that looks at 40 different data shapes to determine which event to publish and which transformation to perform. As someone who's found themselves in that hole before, believe me when I say that is a nightmare situation.
Let's take it a step further. A business operation could (and probably does) affect multiple DynamoDB records. When I publish my newsletter, I create a DynamoDB record with a short TTL for each email address that was successfully sent to. If I am processing entity changes in a stream handler, I'd be up to my ears in duplicate newsletter.sent events because I'm working on individual records and not an aggregate operation. That ultimately leads to workarounds with atomic counters and lastAction flags that make it overly complex with little value add.
And most importantly, you don't know what did or didn't publish. Streams retain data for 24 hours, and if your publisher was broken for a day because of an outage or a rogue IAM change, you're out of luck. Even if you opted for something like Kinesis, which has 365 days of data retention, that's not the point. These mechanisms are change data, not the intended effect on the system.
DynamoDB Streams are a wonderful piece of engineering that gives you retries, but not the durability of intent.
What you should do instead
Time for a bit of a mental shift. We need to prioritize domain events. If an event is important enough to publish, it's important enough to persist in a database.
This is where the outbox pattern comes into play. Instead of deriving your domain events from the data change in a downstream handler, you save the event in an outbox table in a transaction along with the data mutation. The transaction guarantees both the domain event and the data change will save together (or neither).
With DynamoDB, that could look like this:
const eventId = ulid();
await ddb.send(new TransactWriteItemsCommand({
TransactItems: [
{
Update: {
TableName: 'newsletter',
Key: marshall({ pk: issueId, sk: 'issue' }),
UpdateExpression: 'SET #status = :sent',
...
}
},
{
Put: {
TableName: 'outbox',
Item: marshall({
pk: issueId,
sk: `EVENT#${eventId}`,
status: 'PENDING',
pendingBucket: 'PENDING', // GSI1 PK, removed on publish
pendingAt: new Date().toISOString(), // GSI1 SK
source: 'newsletter.service',
detailType: 'newsletter.sent',
detail: { issueId, date, idempotencyKey: eventId }
})
}
}
]
}));
Funnily enough, the architecture doesn't change with this pattern. You're still doing Lambda (or whatever compute) to DynamoDB to a stream to another Lambda function. But this time the downstream Lambda function is processing the event you saved, not the mutated data. It doesn't have to infer anything.
AWS does a fantastic job in their transactional outbox design pattern docs showing you how to setup and implement this pattern yourself. So I won't go into more of the code here. But I will address some questions I had the first time I built this.
What do I do with an event after it's published?
After you publish an event in your stream handler, it's up to you to either delete it from the outbox or mark it as published. Personally, I prefer to mark it as published and set a 30 day TTL. This means in the stream handler, we're hitting the dual write problem again because we are both publishing an event and updating the outbox with a published status.
This one is ok though 😉 and it brings up an important point we haven't covered yet. Idempotency is crucial for success with this pattern. The outbox pattern trades duplicate delivery in exchange for never losing durable content. If your publisher fails to send the event or update the database with a published status, that's fine! Do it again. The important part is that the intent wasn't lost.
Can I replace the stream handler function with EventBridge Pipes?
Fewer pieces, easier system to maintain, right? In this instance, the Lambda function is the better solution over Pipes. If you were only publishing an event to EventBridge, you could easily filter the stream to the relevant outbox records and forward it straight to a bus. But because you also need to write back to DynamoDB to update the status of the event, the function wins. A pipe can only perform a single operation (outside of enrichments and transformations), but a function can perform many.
Does this guarantee ordering?
Nope. The outbox pattern is designed for correctness only. Eventually consistent state across a distributed system. This does not tackle other hard distributed system problems like ordering or exactly-once delivery. Quite the opposite in fact. While Streams do process change data in the order they happen, it's per shard (not globally), and you can't guarantee ordering when it comes to retries. And as soon as you drop an event on EventBridge, you lose any sort of ordering.
How does this guarantee events aren't skipped?
If you implemented this exactly as described, it wouldn't. It gives you the normal retry path you're already used to, but still has that 24-hour clock because of the stream data retention. The important piece that's missing is a scheduled reconciler. It's usually another Lambda function that queries a sparse GSI for anything still pending from more than five minutes ago, which is long enough that the stream handler has had its shot, and republishes it. So your full flow would look something like this:
Remember this
There is no clever workaround to the dual write problem. You can't order the calls in a way that mitigates the risk, you can't retry around it, and you can't/shouldn't move it into a stream handler and hope.
But you can stop having two writes. Well, two writes, but one transaction. 😅
Remember, what we're solving with the outbox pattern is a correctness problem. If someone asks "how can you guarantee the system is eventually consistent, even under backpressure?" this is what you reach for. It took me years to arrive here naturally and I've been accidentally building holes in my systems longer than I'd care to admit.
This pattern does make your architecture more complicated, but the trade-offs are often worth it, especially at scale. In exchange for 2x the WCUs, you get a solid way to store and act upon intent in your system. Anyway, take a look at how you're publishing events and see if there are any unexplained missing events in your system. If there are, you know what to do. 😉 That's what I did and you should be reliably getting your newsletters from now on.
Happy coding!
