One Crypto Swap, Seven Failures: Learning Temporal Through Failure

Building a crypto swap that survives real-world failures — and using each failure to understand what Temporal actually does.

A crypto swap is easy to draw.

A user wants to exchange 10,000 USDC for BTC. We get a quote, wait for the funds, execute the conversion, send BTC, and wait for the payout transaction to confirm.

If every dependency responds, every process stays alive, and every message arrives exactly once, there is not much architecture to discuss.

But those assumptions are precisely what a distributed system cannot make.

The quote provider can return 500. The user can disappear for half an hour. A blockchain RPC can go down. The exchange can execute a trade while its HTTP response gets lost. The process coordinating the swap can crash. And a deployment can change the workflow code while thousands of old swaps are still running.

This is the question I want to use to explore Temporal:

Can we build one crypto swap that keeps its business intent intact as we deliberately break the world around it?

Not every possible failure. Temporal cannot undo a confirmed Bitcoin transaction, fix wrong business logic, or force an external exchange to stay online. The more interesting question is where Temporal's guarantees begin — and where they stop.

The swap in this article is intentionally conceptual. It is not a production exchange, custody, liquidity, or blockchain architecture. The point is to take one understandable business operation and use it as a lens for learning durable execution.

The hidden system behind a simple workflow

Without a workflow engine, the usual instinct is to persist the current state ourselves.

The swap gets a status. We add retry counters. We need a scheduler for deadlines. A queue moves work between services. A recovery job finds transactions that stopped moving. A dead-letter queue catches messages that repeatedly fail. Idempotency keys protect side effects. Operators need a way to inspect and repair stuck swaps.

None of these mechanisms is inherently bad. In many systems they are exactly the right tools.

The problem is that, little by little, we have built a second system whose job is to remember what the first system was trying to do.

This is where Temporal becomes interesting.

Temporal's central idea is durable execution. A Workflow describes the long-running business process. Temporal records the execution as an ordered Event History. Workers execute Workflow Tasks and Activities, but the lifetime of the business process is no longer tied to the lifetime of one worker process.

That distinction is the foundation for everything that follows.

Temporal's documentation describes the Event History as the record that allows a Workflow to be reconstructed after failure. When a Workflow needs to resume, Temporal does not simply deserialize an in-memory snapshot. The Workflow code can be replayed against its recorded history until the previous state is reconstructed. (Temporal: Workflows)

That sounds abstract, so let's start breaking our swap.

Failure #1: the quote provider returns 500

Our first dependency is a quote service. It is also our first failure.

The request fails with HTTP 500.

A second attempt fails too.

The third succeeds.

In Temporal, work that talks to the outside world is typically modeled as an Activity. Activities are where network calls, database writes, and other side effects belong. They are allowed to fail. Temporal can apply a Retry Policy and schedule another Activity attempt after a failure. By default, Activities are retried with increasing delays unless configured otherwise. (Temporal: Retry Policies)

This sounds like a small feature. After all, anyone can write a retry loop.

But a retry loop inside a process only works while that process exists. Once retries are part of the durable execution model, the question changes from "how do I retry this HTTP request?" to "what should this business operation do while a dependency is temporarily unavailable?"

That is a more useful level of abstraction.

It also immediately forces us to think about policy. Which errors are retryable? How long should we back off? When should we give up? What happens after the final attempt? A Retry Policy does not remove those decisions; it gives them a durable place to live.

Failure #2: the user does nothing

The quote is shown, a deposit address is created, and then the user goes for lunch.

Nothing happens for thirty minutes.

In a conventional request/response application, this is an awkward kind of work because there is no useful thread to keep alive. We normally persist a deadline somewhere and arrange for another process to notice it later.

A Temporal Workflow can instead create a durable Timer and wait.
The important word is durable. A Temporal timer is persisted. It is not a sleeping Java thread. The Java SDK documentation explicitly notes that Workflow.sleep() is resource-light and that timers can survive worker or Temporal service downtime and continue when the system is available again. (Temporal: Java Timers)

The swap can now express a business rule such as:

Wait up to 30 minutes for funding. If no valid deposit arrives, expire the swap.

Time has become part of the workflow instead of infrastructure wrapped around it.

Failure #3: the deposit arrives from somewhere else

Five minutes later, a blockchain watcher sees the USDC deposit.

This event did not originate from the Workflow. It came from the outside world, asynchronously, at an unpredictable time.

This is exactly the kind of interaction Temporal models with Signals.

Signals are asynchronous write messages sent to a running Workflow. They can change the Workflow's state or affect its control flow. Temporal also has Queries for reading Workflow state and Updates for synchronous, tracked writes, but our deposit notification fits the Signal model naturally. (Temporal: Workflow Message Passing)

Notice what disappeared from the mental model.

We are no longer asking which pod is "waiting for the deposit." There does not need to be one. The business process is waiting, not a particular machine.

That difference is subtle, but it is one of the reasons durable execution can simplify long-running operations.

Failure #4: kill the worker

Now we reach the experiment that makes Temporal click for many people.

The deposit has been confirmed. The Workflow is moving toward the trade. Then the worker process disappears.

Kill the container. Restart the host. Lose the VM. It does not matter which failure we imagine; the interesting part is that the process holding the Workflow's in-memory state is gone.

What happens next?

The tempting explanation is that Temporal "continues from the line where the worker crashed." That is useful intuition, but technically misleading.

Temporal reconstructs Workflow state through replay.

The Event History contains what has already happened: the Workflow started, the quote Activity completed, the deposit Signal arrived, an Activity was scheduled, and so on. A worker can run the Workflow code again and match the commands it produces against the existing history. Recorded Activity results are reused during replay instead of executing those Activities again. Once replay reaches a point for which there are no recorded events, execution can move forward again. (Temporal: Event History walkthrough for Java)

There is an important distinction here between a Workflow and an Activity.

Workflow replay is about reconstructing deterministic orchestration state. An in-flight Activity is different. If a worker disappears after receiving an Activity Task, Temporal does not magically know at that instant what happened inside the process. Activity loss is detected using timeouts; if the Activity times out and its Retry Policy allows another attempt, Temporal schedules a new one. (Temporal: Activity Execution)

That detail becomes critical in our next failure.

Failure #5: the trade happened, but the response disappeared

The exchange receives our request to convert USDC to BTC.

It executes the trade successfully.

Then the network connection dies before our Activity receives the response.

From the exchange's perspective, the operation succeeded.

From our perspective, it timed out.

Should Temporal retry the Activity?

If the exchange treats a second request as a new trade, a blind retry could execute the conversion twice. Temporal cannot inspect an opaque external system and infer whether the first side effect happened.

This is the point where the phrase "exactly once" needs care.

Temporal recommends making Activities idempotent: executing the operation multiple times should have the same effective result as executing it once. (Temporal: Activity Idempotency)

For our swap, that could mean every trade request carries a stable business idempotency key such as the swap ID. If the first request succeeded but the response was lost, a retry with the same key should return the result of the original trade rather than create a second one.

The broader lesson is more important than the crypto example:

Durable orchestration does not make external side effects exactly-once. It gives you a reliable place to coordinate retries; your side-effect boundary still has to be designed safely.

That is a feature of the mental model, not a weakness to hide. A useful Temporal article should make this boundary explicit.

Failure #6: we converted the money, but cannot deliver the BTC

Suppose the USDC deposit is confirmed and the conversion to BTC succeeds.

Then the payout step fails permanently.

Now what?

We cannot pretend the earlier steps never happened. The service owns BTC that belongs economically to the user, while the user has not received it.

This is where compensation enters the story.

The Saga pattern models a multi-step transaction as a sequence of operations with compensating actions that can be invoked if later work cannot complete. Temporal documents Saga-style compensation as a distributed transaction pattern and recommends that compensations themselves be idempotent. (Temporal: Saga Pattern)

In our fictional service, the correct compensation might be to credit the user's internal BTC balance, return funds where possible, route the case to manual review, or keep retrying a recoverable payout path.

The exact answer is deliberately domain-specific.

A confirmed blockchain transaction cannot be rolled back because our Workflow changed its mind. Temporal makes the decision and execution of compensation durable. It does not make irreversible operations reversible.

That distinction is one of the most useful ways to think about Sagas.

Failure #7: we deploy V2 while V1 swaps are still running

Our service is successful. Thousands of swaps can live for minutes or hours.

Then we deploy a new version of the Workflow.

V1 looked like this:

Quote → Deposit → Swap → Payout

V2 introduces a new step:

Quote → Risk Check → Deposit → Swap → Payout

Normally, deploying new code means future requests run the new code. With replayable long-running Workflows, the situation is more interesting: old executions may later be replayed using code from a newer deployment.

Replay only works if Workflow code remains compatible with the history it is replaying. Temporal describes a Workflow as deterministic when the same input and history lead it to produce the same sequence of commands. A code change that causes replay to expect a different command sequence can break that assumption. (Temporal: Workflow Definition and determinism)

Temporal provides Worker Versioning to help deploy new Workflow code safely. Its current documentation recommends Worker Versioning as the default approach for safe Worker deployments, allowing Workflows to be pinned to compatible Worker Deployment Versions. (Temporal: Worker Deployments)

This is the point where Temporal stops looking like "retries plus a state machine" and starts looking like a different programming model for long-running business processes.

The fact that code may be replayed changes what code is allowed to do. Non-deterministic choices such as reading arbitrary local time, generating randomness directly, or changing command-producing control flow need Temporal-aware handling.

Durability has constraints. That is the trade.

So what did Temporal actually give us?

We began with a seven-step crypto swap and kept changing only one thing: the world around it became less cooperative.

The quote API failed. The user took too long. A deposit arrived asynchronously. A worker died. An exchange response disappeared after a side effect. A payout became impossible. New Workflow code was deployed while old executions still existed.

Each failure exposed a Temporal concept:

Quote API returns 500 → Activities and Retry Policies
User does not fund the swap → Durable Timers
Deposit arrives asynchronously → Signals
Worker disappears → Event History and Replay
Trade result becomes uncertain → Idempotency
Forward progress becomes impossible → Saga / Compensation
Workflow code evolves → Determinism and Versioning

And that gives us a more precise definition of Temporal's value.

It is not primarily that Temporal lets us draw a workflow. We could draw a state machine ourselves.

It is not primarily that Temporal retries requests. We could build a retry library ourselves.

The interesting part is that the execution of the business process becomes durable. Waiting, retries, external events, progress, recovery decisions, and compensation can all participate in one long-lived execution model whose state is not tied to one application process.

What Temporal does not do

There is a dangerous version of the Temporal story where the conclusion becomes: "Put your code in a Workflow and failures go away."

That is not what happened in our swap.

Temporal helped preserve the execution of the process. It did not eliminate the distributed-system boundaries around it.

Temporal can help us remember that a swap exists after a process crash. It can durably coordinate retry policy. It can wait without holding an application thread. It can receive external messages. It can replay Workflow state. It can help us encode compensation and safely evolve long-running Workflow code.

But it cannot undo a confirmed blockchain transfer. It cannot determine whether an arbitrary external API performed an operation if that API provides no way to resolve the ambiguity. It cannot make a non-idempotent side effect safe by retrying it. And it cannot choose the correct business compensation for us.

Those remain architecture and domain problems.

That boundary gives me a useful heuristic for when Temporal becomes interesting:

Temporal earns its complexity when the lifetime of a business operation is meaningfully longer — and less reliable — than the lifetime of the process currently executing it.

A CRUD endpoint that reads a row and returns JSON probably does not need it.

A multi-step process that crosses services, waits for humans or external events, survives deployments, performs expensive side effects, and must not silently disappear is a much stronger candidate.

Our crypto swap was useful not because crypto is special, but because it puts all of those forces into one small picture.

The happy path took one diagram.

The interesting architecture began when we started breaking it.