What may degrade, and what must fail fast

Six services sit between a buyer clicking "buy" and a vendor eventually getting paid: auth, gateway, catalog, inventory, order, and now settlement. Any one of them can be slow, restarting, or fully down at any moment — that's the normal condition of a distributed system, not an incident you fix once and move past. The question worth writing down isn't "how do we prevent that." You can't, fully. It's: when it happens, which requests should degrade to a worse-but-safe answer, and which must refuse outright rather than guess?

The instinct most people reach for first is a single global rule — either "always degrade gracefully" or "always fail fast." Both are wrong the moment you apply them uniformly across services that don't carry the same kind of risk. I found the actual criterion isn't "how important is this service." It's whether the business cost of guessing wrong is reversible.

The criterion, not the checklist

A service whose worst-case wrong guess is "the buyer sees something slightly stale" may degrade — the guess self-corrects the moment the dependency comes back, and nothing irreversible happened while it was wrong. A service whose worst-case wrong guess is "we sold something we don't have" or "money moved that shouldn't have" must fail fast, because there is no UPDATE statement that un-ships a package or un-spends a vendor's payout. That's the whole rule. Everything below is just applying it, service by service, to real code that's actually running.

catalog down → degrade

The storefront can serve a cached snapshot of what it last saw, or an empty result for an uncached query, and let the buyer retry. Worst case: a product looks unavailable that's actually fine, or a price is a few seconds stale. Both are cosmetic and self-correct the instant catalog comes back. Nothing about "can't browse right now" commits the business to anything.

inventory down → fail fast

order.create()'s call to inventory's reserve endpoint is synchronous, and it has no fallback path — deliberately. If the fallback were "let the order through and reconcile inventory later," the failure mode is overselling the last unit of a scarce SKU to five buyers at once, discovered only when four of them can't be shipped what they paid for. Checkout returning a 503 and telling the buyer to retry is strictly better than checkout succeeding and lying. This is the same conditional-update invariant I wrote about in the inventory-reservation piece — WHERE available >= ? either commits or it doesn't, and there's no version of "degrade gracefully" that doesn't mean "sell something you can't deliver."

settlement down → degrade to zero visible effect

This is the newest row in the table, and the one that only exists because M3 added a sixth service. order never calls settlement synchronously — the only coupling is order publishing order-paid, sub-order-delivered, and sub-order-refunded to Kafka and moving on. Settlement being down means those events queue up unconsumed. Checkout, payment, and fulfillment are entirely unaffected, because nothing on the buyer path is waiting for settlement to acknowledge anything. The cost lands entirely on vendors getting paid later than usual — bounded and recoverable, because Kafka retains the backlog and settlement's own idempotent consumer means catching up on a day of queued events produces the same end state as processing them on time. I didn't just argue this — the M3 gate exercises it directly: place an order, split it, pay, mark it delivered, and only afterward bring the billing sweep around to consume the resulting events. Nothing in the buyer-facing steps has ever depended on settlement being reachable, by construction, not by the gate happening to get lucky with timing.

auth down → degrade for issued tokens, fail fast for new ones

The gateway validates JWTs locally against a cached JWKS — a request carrying a still-valid signed token is authorized without calling auth at all, so a buyer mid-session keeps shopping through an auth outage with zero visible effect. A brand-new login does have to reach auth, and that has to fail rather than accept a token it can't verify. Skipping that check isn't a degradation choice, it's a decision to stop checking signatures — a security regression wearing a resilience costume.

The table, for anyone paging through an incident

The value of writing this down isn't the table itself — it's that "which side of this table a service belongs on" is a property of what the wrong guess costs, not of how central the service feels in an architecture diagram. Add a seventh service later, and the question to ask is the same one this table already answers: can a wrong-guess response be undone by a later correction, or does it commit the business to something — stock shipped, money moved — that can't be clawed back cleanly? The answer decides the failure mode before a single line of fallback code gets written, not after an incident reveals it was wrong.

It's also worth naming what this framework is not saying. It's not "settlement doesn't matter" — money not reaching vendors on time matters a great deal to the vendors waiting on it. It's that the recovery is cheap and bounded (drain the backlog; reconciliation catches anything that still doesn't add up), so it belongs in the "degrade" column even though the thing it's protecting is high-stakes. Reversibility, not stakes, is the axis this sorts on.

Where a circuit breaker fits, and where it doesn't

Resilience4j sits at the gateway, and it's tempting to treat "add a circuit breaker" as a solved, mechanical step once you've written a resilience ADR. It isn't, because a circuit breaker's failure-rate threshold and open-state duration are business judgments wearing config-file syntax, not defaults you tune once and forget. Set the threshold too sensitive and you trip the breaker on an inventory service having one slow GC pause, converting a 20ms blip into checkout being unavailable for the whole open-state window — you've just built your own worse incident on top of a fake one. Set it too lax and the breaker never opens before a genuinely struggling dependency has already caused the actual harm you built the thing to prevent.

The number that should set that threshold isn't a Resilience4j tutorial's example config — it's the answer this ADR already worked out for each dependency. A breaker in front of inventory should trip fast and stay open long enough that a struggling instance gets a real chance to recover, because every request that gets through to a failing inventory call is a request that either fails fast (correct) or, worse, times out slowly while a buyer stares at a spinner. A breaker in front of catalog, by contrast, can afford to be far more patient, because a slightly-too-eager open state there just means a few extra buyers see a stale result they'd have seen anyway. Same library, same default config shape, opposite tuning — because the two dependencies sit on opposite sides of the reversibility line this whole piece is about. The circuit breaker doesn't replace the judgment call. It just automates executing whichever judgment call you already made.

This is part of a series on building a multi-vendor commerce platform. The open-source half, stallora-cloud-starter, carries the gateway and auth services this piece describes. Next up: shipping a six-service stack that buyers can actually run on a 6 GB VPS.