Distributed Transactions are one of the hardest problems you can face in any distrubuted systems.
It is not because of the complexity of implementation, but it is all about the mental model change that you go through, going from a single ACID transaction into a distrubuted one.
Inside one database, ACID handles this for you: any step fails, the whole thing rolls back, clean. Spread that order across four microservices and the safety net is gone. No transaction can span four databases.
The instinct is two-phase commit. Don't. 2PC holds locks across every service while it waits for all of them to vote yes, so one slow participant freezes the whole set. Most managed and NoSQL databases won't support it anyway.
That's the saga: a chain of local transactions, each with a compensating action for when a later step fails.
Here's what most people miss. A saga gives you A, C, and D. It quietly drops the I. No isolation across services means two sagas running at once can corrupt each other: lost updates, dirty reads, non-repeatable reads. Real anomalies, in production, that no framework hides for you.
And "compensation" is not rollback. Committed data doesn't un-commit. You run a new transaction that makes it right: a refund, a cancellation, a restock.
Two ways to coordinate the same saga.
Choreography: services react to each other's events, no central coordinator. Clean with 3 services. A nightmare to trace at 3am when it's 9.
Orchestration: one brain (Step Functions, Temporal) tells each service what to do next. More moving parts, but you can see the flow and debug it.
The fix for the missing isolation is boring and it works: a status column. PENDING, then CONFIRMED or CANCELLED. A semantic lock so nothing downstream acts on half-finished state.
I put the whole thing into one video. Why 2PC is a dead end, choreography vs orchestration running the same saga, the isolation anomalies that bite at scale, and one food-order example carried across four services start to finish.
