Scaling Realtime Event Delivery for 10,000 Reconnecting Delivery Tracking Maps

For realtime release compatibility in a delivery tracking map, scale event delivery with a durable, ordered log per delivery and treat every browser connection as a disposable projection of that log. Presence can guide fan-out and capacity planning, but it must never decide whether a location update exists.

Short answer: release compatibility comes from versioned envelopes, resume cursors, and an explicit resync path; scaling comes from partitioning by delivery ID and coalescing map updates at the edge, not from trusting a long-lived connection to carry every event exactly once.

This decision targets an e-commerce tracking experience in which a shopper may open a map, lose connectivity in a tunnel, return on another network, and also join a delivery-specific support chat room. The deciding constraint is presence accuracy: an online indicator is useful only when its expiry rules are understood, while the delivery state must remain correct even when that indicator is late.

A green dot isn't a commit log.

How should realtime release compatibility scale event delivery in a delivery tracking map?

Separate the system into three contracts: durable delivery state, transient room presence, and the connection used to move updates. The first contract owns truth. The second answers a narrower question: which sessions have renewed a lease recently enough to be considered reachable? The third may disappear at any point and should be replaceable without changing either of the other two. For each delivery, append an event with a monotonically increasing sequence within that delivery's partition. The client persists the last applied sequence and includes it when reconnecting. If retained events cover the gap, the server replays them in order; if they don't, the server returns a fresh snapshot plus its sequence. This is at-least-once delivery with idempotent application, which means duplicates are ordinary and gaps are detectable. It does not promise global order across unrelated deliveries, because a tracking map doesn't need it and the coordination cost would buy no visible correctness. The release envelope should carry an event type and schema version alongside the delivery ID, sequence, and timestamp. During a rolling release, producers emit a version that both old and new consumers understand; readers ignore additive fields they don't recognize, and an incompatible semantic change gets a new event type or a deliberate version transition. Don't silently reuse a field with a different meaning. That turns a deployment detail into corrupted state, and no reconnect algorithm can repair it.

These are the invariants worth writing into the architecture decision record:

• A sequence is unique and increasing within one delivery stream.
• A cursor advances only after the client has applied an event.
• A snapshot and its cursor describe the same logical point.
• Presence expires unless renewed; disconnect callbacks are hints, not proof.
• An older map update cannot overwrite a newer applied sequence.

The failure boundaries follow directly. A dropped connection can delay a view but cannot erase history. A duplicated event can spend bandwidth but cannot move state backward. A stale presence lease can briefly overcount a room, but it cannot alter package location or chat history. If a new release changes an envelope incompatibly, the consumer rejects that version and requests a snapshot rather than guessing.

The option comparison

The transport choice matters, but less than the recovery contract. The table compares architecture shapes rather than brands; any implementation still needs load tests against its own payload distribution and reconnect pattern.

There is no universal winner. Snapshot polling is a sensible default when positions change slowly and operational simplicity matters more than immediacy. A direct peer connection can be valid for a small support interaction, and the W3C WebRTC Recommendation specifies the browser peer-connection model, but peer connectivity alone does not define durable replay, authoritative presence, or event-version migration. Those remain application contracts.

I'm not sure what reconnect burst a particular storefront will see; session length, mobile network behavior, and campaign traffic determine it. Resolve that uncertainty with a test that disconnects a measured share of clients, advances delivery streams while they are away, then reconnects them within the same short window. The pass condition is state convergence at the newest sequence, not merely a successful handshake.

Critical path and failure handling

The critical path below is deliberately transport-neutral Python. store.read_after returns ordered events for one delivery when the cursor is still retained, while store.snapshot returns a state paired with the sequence that produced it. Authentication and network framing sit outside this function; mixing them into replay logic makes compatibility tests harder to isolate.

from dataclasses import asdict, dataclass
from typing import Any, Protocol


@dataclass(frozen=True)
class Event:
    delivery_id: str
    sequence: int
    event_type: str
    schema_version: int
    payload: dict[str, Any]


class DeliveryStore(Protocol):
    def read_after(self, delivery_id: str, sequence: int) -> list[Event] | None: ...

    def snapshot(self, delivery_id: str) -> tuple[dict[str, Any], int]: ...


def resume_delivery(
    store: DeliveryStore,
    delivery_id: str,
    cursor: int,
    accepted_versions: set[int],
) -> dict[str, Any]:
    events = store.read_after(delivery_id, cursor)
    if events is None:
        state, snapshot_sequence = store.snapshot(delivery_id)
        return {
            "kind": "snapshot",
            "delivery_id": delivery_id,
            "sequence": snapshot_sequence,
            "state": state,
        }

    compatible = [
        event for event in events if event.schema_version in accepted_versions
    ]
    if len(compatible) != len(events):
        state, snapshot_sequence = store.snapshot(delivery_id)
        return {
            "kind": "snapshot",
            "delivery_id": delivery_id,
            "sequence": snapshot_sequence,
            "state": state,
        }

    return {
        "kind": "events",
        "delivery_id": delivery_id,
        "events": [asdict(event) for event in compatible],
    }

Keep client application equally strict: discard an event whose sequence is at or below the applied cursor, apply the next expected sequence, and request resync when a gap appears. Never manufacture the missing location by interpolation as a recovery mechanism. Interpolation may animate between two known points, but it cannot establish where the authoritative stream says the package is.

Presence needs a different clock. A session joins a delivery room with a short lease, renews it while active, and disappears after expiry; a clean disconnect may remove it sooner, but correctness cannot depend on receiving that signal. Count sessions and users separately, since one shopper can have a phone and laptop open. For support chat, attach message durability to the room's event history and reserve presence for labels such as online, recently active, or unknown.

Binary certainty is usually dishonest here.

Observe four separate outcomes: connection attempts, resume results, replay depth, and resyncs. Aggregate map frames or connection counts can look healthy while one partition repeatedly falls outside retention. Alert on gap and snapshot rates by release version, then compare old and new consumer cohorts during rollout. Roll back a consumer before removing its supported envelope; remove old schema support only after its client population and retained events have aged out under a documented policy.

Ten thousand connected maps also shouldn't imply 10,000 identical reads for every courier point. Partition by delivery ID, read each new durable event once per serving node, and fan it out to local subscribers. Coalesce intermediate coordinates over a bounded display interval when the UI only needs the newest position, but never coalesce status transitions such as delivered or canceled.

Business events aren't replaceable.

Why the ephemeral room was rejected

An ephemeral room looked attractive because its live path is small: join, broadcast, leave. It was rejected for the tracking map because a shopper who reconnects has no evidence that the current marker follows all prior delivery transitions, and presence churn becomes entangled with state correctness. A network break then forces an improvised full refresh, while a rolling release has no stable cursor or envelope boundary on which to negotiate compatibility.

Still, keep that option where loss is acceptable. Typing indicators, pointer trails, and short-lived map animation hints can use ephemeral broadcast because a missed signal should simply vanish. It is also suitable for a bounded internal display that periodically reloads an authoritative snapshot and makes no claim of continuous history.

The catch for the durable-log design is real operational ownership: retention windows, partition hotspots, snapshot consistency, and schema retirement all need explicit policies. Stick with polling when update volume is modest, users tolerate the polling interval, and the team cannot operate replay storage confidently. Choose peer communication only for small rooms whose membership and recovery semantics are bounded. The decision is about failure ownership, not transport fashion.

Release test and deployment record

A release is compatible only after mixed-version testing proves it. Run an old producer with a new consumer, then a new producer with an old consumer, and include disconnects on both sides of every emitted event. Test duplicate sequence 41, a gap from 41 to 43, a cursor older than retention, two simultaneous sessions for one user, and a presence lease that expires without a disconnect notification. Those cases are small enough to automate and specific enough to diagnose.

Record the accepted schema versions, retention window, snapshot source, partition key, lease duration, and rollback order with the release. Avoid a single “realtime latency” dashboard: it hides whether time was spent before persistence, in fan-out, during client application, or waiting for recovery. Your mileage may vary on thresholds, so derive them from the tracking product's freshness promise and measured traffic rather than copying a generic target.

One rule closes the record: durable delivery truth survives every connection, while presence remains an expiring estimate. With that boundary intact, transports and deployment versions can change without asking a reconnecting map to guess what it missed.

References

• W3C, “WebRTC: Real-Time Communication in Browsers”: https://www.w3.org/TR/webrtc/