9 min read · warehouse execution system, global retail & manufacturing
A few years ago I worked on a warehouse execution system (WES) — the software that tells a warehouse what to pick, pack, and ship next, in real time, for retail and manufacturing clients moving somewhere between 5 and 10 million requests a day. Orders come in, inventory gets allocated, pick tasks get generated, shipments get confirmed — and every one of those steps talks to the others through messages.
The system ran on ActiveMQ. It had worked for years. But at that volume, two problems had stopped being rare edge cases and started being a weekly occurrence: duplicate order processing, and failures nobody noticed until a customer did. This post is about what we actually changed — not just "we moved to NATS," but the specific patterns that made the new system trustworthy instead of just newer.
The problem wasn't really the broker
It's tempting to blame ActiveMQ itself. That's not quite fair. The real problem was what had been built around it over the years: a direct, synchronous-feeling coupling between "write to the database" and "publish a message," with no shared transaction between the two. At low volume, that gap almost never showed up. At 5–10M messages a day, it showed up constantly, in two specific ways:
- Dual-write inconsistency. A service would commit an order update to the database, then publish an event. If the process crashed between those two steps — or the publish failed silently — the database and the message stream disagreed about what had happened. Sometimes an event fired twice for the same order. Sometimes it didn't fire at all.
- No systematic way to detect "this already happened." Consumers processed whatever arrived. If a message got redelivered — which happens by design in most messaging systems, because "at-least-once" is the realistic guarantee, not "exactly-once" — the consumer had no way to know it was seeing the same order for the second time.
Put together, that's how you get an order picked twice, or a shipment confirmation that silently vanishes. Nobody had done anything "wrong" — the system had just been built for a lower-stakes version of itself, and it hadn't been revisited as the stakes went up.
Why NATS, specifically
NATS JetStream wasn't chosen because ActiveMQ is "bad" — it was chosen because the operational model fit this specific problem better: lighter-weight persistence, simpler clustering, and a consumer model that made explicit acknowledgment and replay straightforward to reason about. That mattered less than what we built on top of it — but it's worth saying plainly: the broker swap was the easy 20% of this project. The patterns below were the other 80%.
Pattern 1 — Outbox, so the database and the message stream can't disagree
The fix for dual-write inconsistency is the Outbox pattern: instead of writing to the database and then publishing a message as two separate operations, you write the business change and a row representing "an event needs to be published" in the same database transaction. A separate relay process reads unpublished rows and publishes them, then marks them published.
CREATE TABLE outbox_event (
id UUID PRIMARY KEY,
aggregate_id VARCHAR(64) NOT NULL,
event_type VARCHAR(64) NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
published_at TIMESTAMPTZ
);
This sounds almost too simple to matter. It matters a lot. It moves the "did this actually happen" question from "check two systems and hope they agree" to "check one row in one transaction." That single change removed an entire category of incident.
Pattern 2 — Idempotent receivers, because redelivery is normal, not a bug
Once you accept that messages can and will be delivered more than once — which is the honest default for almost any distributed messaging system — the consumer has to be able to answer "have I already handled this?" before doing anything with side effects.
@Transactional
public void handle(OrderEvent event) {
if (processedEventRepo.exists(event.getEventId())) {
return; // already handled — safe to ignore
}
applyOrderUpdate(event);
processedEventRepo.markProcessed(event.getEventId());
}
The mechanics here are almost boring — a table of processed event IDs, checked before any state change, updated in the same transaction as the change itself. What isn't boring is how much it changes the character of incidents: instead of "why did this order get picked twice," the failure mode becomes "a message was redelivered and correctly ignored," which is invisible to the business and shows up as a log line instead of a support ticket.
Pattern 3 — Saga, for the things no single transaction can cover
Order processing touched inventory, picking, and shipment — three services, three databases, no single transaction spanning all of them. That's exactly the situation the Saga pattern exists for: model the multi-step process as a sequence of local transactions, each with a defined compensating action if a later step fails.
Concretely: if inventory allocation succeeds but pick-task generation fails, the saga triggers a compensating "release allocation" step rather than leaving the system in a half-committed state. None of this is exotic — it's the standard answer to distributed consistency without two-phase commit — but it only works if step 1 (idempotency) and step 2 (Outbox) are already solid. Saga compensations that fire against a system that can't tell "processed" from "not yet processed" just create a second category of duplicate work.
Pattern 4 — Retries with backoff, and a dead-letter queue with an actual runbook
The last piece was making failure explicit and bounded, instead of infinite or silent.
consumer:
durable_name: order-service
ack_policy: explicit
max_deliver: 5
backoff: ["1s", "5s", "30s", "2m", "10m"]
dead_letter_subject: "orders.dlq"
A message gets a bounded number of attempts with increasing backoff. If it still fails, it goes to a dead-letter subject — and this is the part that's easy to skip and expensive to skip: a dead-letter queue that nobody monitors is just a slower, quieter way of losing messages. We paired the DLQ with an actual on-call runbook: what triage looks like, who gets paged, and what "safe to replay" versus "needs a human" looks like for each event type.
Before → After, in short:
- Before: App writes to DB, then separately publishes to ActiveMQ (two uncoordinated steps) → consumer processes whatever arrives, with no way to detect a repeat → failures are silent until someone downstream notices.
- After: App writes to DB and an Outbox row in one transaction → a relay publishes to NATS JetStream → an idempotent consumer checks a processed-events table before writing → failures beyond the retry budget go to a dead-letter subject with an actual runbook attached.
(There's a diagram of this in the full post on my site — link at the bottom.)
Migrating without a big-bang cutover
None of this shipped as one release. The order that mattered:
- Outbox first, on the existing ActiveMQ setup. This alone fixed the dual-write problem before touching the broker at all, and proved the pattern under real load.
- Idempotent receivers next, also still on ActiveMQ. This removed the duplicate-processing risk independently of the broker migration.
- NATS introduced consumer-by-consumer, running in parallel with ActiveMQ for a transition window, comparing outputs before cutting each consumer over.
- ActiveMQ retired last, once every consumer had been running clean on NATS for long enough to trust it.
Doing it in that order meant every step was independently safe to roll back, and the riskiest-sounding part of the project — "replace the message broker" — ended up being the calmest step, because the parts that actually caused incidents had already been fixed before the broker changed at all.
What I'd tell someone starting this today
- Idempotency isn't optional at this point of the process — it's the thing that turns "redelivery" from an incident into a no-op.
- Fix the dual-write problem before you touch the broker. Outbox is boring, unglamorous, and it's the highest-leverage change on this list.
- A dead-letter queue without a runbook is a to-do list nobody reads. Decide who gets paged and what "replay" means before you need it at 2am.
- Migrate the parts that reduce risk before the part that looks risky. The broker swap was the least stressful part of this project — because by the time it happened, the system could already tell the difference between "new" and "already handled."
If your system has a similar shape — high message volume, a broker that's shown its age, or a nagging feeling that "duplicate" incidents keep coming back in slightly different clothes — this is exactly the kind of thing a short audit is good for: not a rewrite, just an honest map of where the Outbox, idempotency, and retry gaps actually are before you touch anything.
Full post with architecture diagram: Link post
*Book a 20-min call: Book a call
