you've got an event‑sourced aggregate that emits a UserDeleted event, and several downstream services react by cleaning up related rows in their own stores. the cleanup runs in background workers, so a second request that references the same user can slip through the cracks and hit a foreign‑key violation.
the race condition
in a typical setup each service subscribes to the UserDeleted topic and issues a cascade delete against its own tables. because the subscription processing is asynchronous, a request that arrives after the UserDeleted event is published but before the worker finishes will see stale data. the DB then throws an error, and the client gets a 500 instead of a graceful "not found".
a deterministic approach
the trick is to make the delete operation idempotent and observable before the request reaches the service. one pattern is to store a tombstone flag in a shared cache (e.g. Redis) as part of the same transaction that publishes the event. workers check the flag before proceeding, and request handlers read it to short‑circuit.
// publish + tombstone atomically
await redis.multi()
.set(`tombstone:user:${userId}`, '1', 'EX', 300) // 5 min TTL
.publish('user.deleted', JSON.stringify({ userId }))
.exec();
workers then do:
const flag = await redis.get(`tombstone:user:${userId}`);
if (!flag) return; // nothing to clean up
await db.transaction(async trx => {
await trx('orders').where({ userId }).delete();
await trx('profiles').where({ userId }).delete();
});
await redis.del(`tombstone:user:${userId}`);
trade‑offs
- latency vs consistency – adding the cache lookup adds a few milliseconds to every request, but it eliminates sporadic DB errors.
- cache eviction – the TTL must be long enough to cover the slowest worker, otherwise a stale request could still hit the DB.
- failure handling – if the worker crashes after deleting rows but before clearing the tombstone, subsequent requests will be rejected even though the data is gone. a periodic reconciler that scans for orphaned tombstones can mitigate this.
- testing – simulate the race by issuing a delete and a read in parallel within integration tests; assert that the read returns a 404 rather than a DB exception.
when to apply
if your service experiences occasional 500 spikes after bulk deletions, or if you already use a message broker for domain events, the tombstone pattern is a low‑overhead way to enforce eventual consistency without locking the DB.
anyone using a similar approach with different stores (e.g. DynamoDB streams) or who prefers optimistic concurrency controls instead? how do you keep the delete path safe without sacrificing throughput?
