Offline-First for a Mobile Service Van: Designing Field Sync That Survives a Parkade
This article is a design exercise, not a case study. Nothing described here has been built, deployed, or operated by KMJ Tire. There is no production system behind these diagrams, no rollout, no incident history, and no measured numbers. What there is: a real service domain — a mobile tire van working Calgary and the surrounding acreages — that makes the constraints concrete instead of hypothetical. Every schema, payload, and snippet below is illustrative, written to be read and argued with, not copied into a repository and run. No code here has been executed or benchmarked against any database.
With that out of the way, here is the problem.
The Domain That Sets the Constraints
A mobile tire van does the same work a fixed bay does, minus the building. Seasonal changeovers, mount and balance, flat repairs, rotations, tread depth checks, oil changes. The difference is where the work happens: a residential driveway in Tuscany, a fleet yard off Barlow, the third level of a downtown parkade, an acreage past Springbank Road where LTE is a rumour. Some of that work is scheduled days ahead; some of it is unplanned roadside work that lands mid-route, and roadside jobs are never in a convenient place for a network.
Connectivity in those places ranges from fine to nonexistent, and the failure is rarely clean. Parkades are the worst case people expect. The worse case people forget is the partial one: two bars, a TCP connection that opens and then stalls, DNS that resolves after eleven seconds, a POST that the server actually processed but whose response never made it back down. Rural stretches out toward Bragg Creek or north past Balzac are simpler — you're just off the grid — and simple is easier to design for than flaky.
The business constraint is the one that matters. The tech cannot stop working. If the tablet refuses to record a tread depth reading because the network is down, the reading goes on a scrap of paper, or into someone's head, and then it is gone. The whole value of the field app is that the record of work gets created at the moment and place the work happens, by the person who did it. Anything that breaks that link degrades the app into a form that people fill out later from memory, which is worse than no app at all because it looks authoritative.
October and November sharpen this. Changeover season in Calgary compresses an enormous amount of routine work into about six weeks, and the van is doing five or six stops a day across wildly different coverage. The system's worst day is also its busiest day. That is the design target, not the sunny afternoon in Bridgeland with full signal.
Offline-First Is a Data Model Decision
Here is the claim I will keep coming back to: you do not add offline support to an application. You either designed for it or you did not, and retrofitting it is close to a rewrite of everything that touches persistence.
The reason is that a conventional client-server app encodes an assumption in its data model so deeply that nobody notices it: the server's row is the truth, and the client is looking at a photograph of it. Every mutation is a request to change that row. The client's job is to display, collect input, send, and re-render whatever comes back. Under that model the client has no opinion. It has no state of its own worth preserving. When the network fails, there is nothing sensible to do except show a spinner and then an error, because the client literally does not know what it means for something to be true locally.
Offline-first inverts that. The device's local database is a first-class replica with authority over the facts it originated. The server is not the source of truth for "the tech observed 4 mm on the left rear at 10:42" — the device is, because that observation happened there. The server is the source of truth for aggregate, cross-device state: which vehicle this belongs to, what the customer was quoted, whether the work order is closed. Sync is the process of reconciling two authoritative-in-their-own-domain stores, not refreshing a photograph.
Once you accept that framing, a pile of decisions falls out of it. Primary keys have to be generatable on the device, so they are UUIDs or ULIDs, not database sequences. Records need to carry their own origin and version metadata. Deletes cannot be row removals, because a removal is invisible to a replica that never saw it. Timestamps become suspicious data rather than reliable ordering. And "save" stops meaning "the server has it" and starts meaning "this is durably recorded locally and will be transmitted."
That last one is a product decision as much as a technical one, and it is the one that most often gets fumbled. If the UI says "Saved" when it means "queued," you have made a promise on behalf of a queue that might never drain. Two states, always: recorded locally, and confirmed upstream. Show both. Field techs are not confused by this — they already understand the difference between writing something down and handing it in.
The Write-Ahead Mutation Log
The core mechanism is a durable, ordered, append-only log of intended changes, living on the device, drained in order by a background worker.
I call it a mutation log rather than a queue because "queue" implies you can drop things off the front and forget them. You cannot. You need the history for debugging, for reconciliation, and for the moment six weeks later when someone asks what the tablet actually recorded.
SQLite is the right substrate on a tablet — it is transactional, it survives process death, and it is available everywhere. IndexedDB works in a browser context with more caveats. What you must not do is hold pending mutations in memory, or in a JSON file rewritten wholesale on every change, because a van tablet dies in ways desktop software never does: battery pulled at -28 in a yard, dropped, OS-killed while backgrounded, wedged in a dock that stops charging.
CREATE TABLE mutation_log (
mutation_id TEXT PRIMARY KEY, -- ULID, generated on device
seq INTEGER NOT NULL, -- monotonic, device-local
entity_type TEXT NOT NULL, -- 'work_order', 'tread_reading', ...
entity_id TEXT NOT NULL, -- UUID, generated on device
op TEXT NOT NULL, -- 'create' | 'append_event' | 'retire'
payload BLOB NOT NULL, -- serialized intent
schema_version INTEGER NOT NULL,
device_id TEXT NOT NULL,
lamport INTEGER NOT NULL,
wall_clock_utc TEXT NOT NULL, -- advisory only, never for ordering
depends_on TEXT, -- mutation_id of a prerequisite
state TEXT NOT NULL, -- 'pending'|'inflight'|'acked'|'poison'
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
acked_at_utc TEXT
);
CREATE INDEX mutation_log_drain ON mutation_log (state, seq);
The seq column is the spine. It is a device-local counter that only ever increases, and it defines the order in which mutations were intended, which is the only ordering the device can vouch for. The drain worker reads pending rows in seq order and stops at the first unrecoverable failure rather than skipping ahead — reordering is how you end up applying a status change to a work order that does not exist yet.
depends_on exists for the cases where strict global ordering is too heavy. If the tech creates a work order and then attaches three tread readings, those readings depend on the creation but not on each other. A drain worker that understands dependencies can parallelize the independent ones and still refuse to run a child before its parent. Most teams should start with strict serial drain and add this only when serial drain is measurably too slow, which for a van doing six stops a day it will not be.
State transitions are worth being explicit about, because ambiguity here produces duplicates:
+------------------ retry (attempts < N) ------------------+
v |
[pending] --> [inflight] --+--> 2xx / 409-duplicate --> [acked] --> (pruned after 90d)
|
+--> 5xx / timeout / offline --> [pending]
|
+--> 4xx validation --> [poison] --> (surfaced to a human)
The subtle transition is 409-duplicate → acked. If the server says "I already have this mutation," that is success, not failure. A design that treats a duplicate rejection as an error will retry forever and eventually poison a perfectly good record.
Client-Generated Idempotency Keys
Every mutation carries a key the device invented before it ever tried to transmit. That key is stable across every retry of that mutation, forever.
This is the single highest-leverage decision in the entire design, and it is nearly free. Without it, the classic field failure is unavoidable: the tech taps to record a completed rotation, the request reaches the server, the server writes it, the response dies somewhere over the parkade ramp. The device sees a timeout. The device retries. Now there are two rotations on the vehicle's history, and the second one is invisible to the tech, who saw one success. Nobody notices until a customer looks at a service record and asks why the same job appears twice.
The fix is that the server keys on the client's identifier rather than deriving its own:
type Mutation<T> = {
mutationId: string; // ULID, minted once, reused on every attempt
entityId: string; // UUID v4, also minted on device
entityType: 'work_order' | 'tread_reading' | 'service_event' | 'photo_ref';
op: 'create' | 'append_event' | 'retire';
schemaVersion: number;
deviceId: string;
lamport: number;
payload: T;
};
async function drainOne(m: Mutation<unknown>): Promise<DrainResult> {
const res = await transport.post('/v1/mutations', m, {
headers: { 'Idempotency-Key': m.mutationId },
timeoutMs: 15_000,
});
if (res.status === 200 || res.status === 201) return { ok: true };
if (res.status === 409 && res.body?.reason === 'duplicate') return { ok: true };
if (res.status >= 400 && res.status < 500) return { ok: false, poison: true, detail: res.body };
return { ok: false, retryable: true };
}
Two properties make this work. The key is minted at the moment of intent — when the tech's finger leaves the screen — not at transmission time, because a key generated per attempt is not an idempotency key, it is a random number. And the key is durable: it lives in the same transactional write that recorded the mutation, so a device that reboots mid-flight resumes with the same key rather than inventing a new one.
The server side is a table and a uniqueness constraint:
CREATE TABLE applied_mutations (
mutation_id TEXT PRIMARY KEY,
device_id TEXT NOT NULL,
entity_id TEXT NOT NULL,
applied_at TIMESTAMPTZ NOT NULL DEFAULT now(),
response_body JSONB NOT NULL
);
Storing the original response body matters more than it looks. When a retry arrives for a mutation already applied, you want to return the same answer you returned the first time, not a bare acknowledgement. The device may have been waiting on a server-assigned identifier or a computed field; giving it a different shape on the retry path means the retry path is undertested and will break in a way nobody reproduces.
Why a Dedupe Window Is Not Enough
The common shortcut is server-side deduplication over a time window: hash the request body plus the user and the endpoint, reject anything identical within five minutes. It is easy, it requires no client changes, and it fails in exactly the situations you built it for.
Start with the window. Five minutes assumes retries happen promptly. A van that starts a job in an underground parkade and does not see signal again until it is back up on 9th Avenue has been offline for forty minutes. Every mutation from that stop replays outside any window you would consider reasonable. Widen the window to a day and you have built an unbounded hash table with a retention policy, which is the idempotency table you were avoiding, minus the correctness.
Then the hashing. Two genuinely distinct mutations can serialize identically. A tech records 6 mm on both front tires within the same minute — same reading, same vehicle, same user, differing only by a position field that a sloppy hash might exclude. Content hashing cannot distinguish "the same event sent twice" from "two identical events," and only the client knows which one it meant.
Finally, the failure mode is silent in the wrong direction. A dedupe window that is too aggressive drops real data with a 200 OK, and the device is satisfied. Duplicates are ugly and visible. Silent loss is neither, and it is the one that costs you a customer's trust when a documented tread measurement turns out never to have existed. Explicit client-minted keys make the intent unambiguous and let the server stop guessing. Dedupe windows are a reasonable defence-in-depth layer behind real idempotency. They are not a substitute for it.
Recording Intent Rather Than State
Now the part that makes conflict resolution tractable, because most conflicts are self-inflicted.
Consider the difference between these two ways of saying the same thing. First, a state mutation:
PATCH /work_orders/e2f1
{ "left_rear_tread_mm": 4.0, "status": "in_progress" }
Second, an event:
{
"eventId": "01J9F2M0Q7S3R8VK",
"workOrderId": "e2f1",
"type": "tread_measured",
"observedBy": "tech-14",
"position": "left_rear",
"valueMm": 4.0,
"instrument": "depth_gauge",
"lamport": 118,
"deviceId": "van-02"
}
The PATCH is a demand: make the row look like this. If two devices issue conflicting PATCHes, the system has no way to reason about them, because the request has thrown away everything about where the value came from. All that survives is a number and an arrival time.
The event is a claim about the world: this person, using this instrument, observed this value at this position. It is immutable. Nothing contradicts it, because it is a fact about an observation rather than an assertion about current state. If a second measurement arrives from a different device, that is not a conflict — it is a second observation, and both are true. Whether the vehicle's current record should show 4.0 or 3.5 becomes a projection question you answer with an explicit rule, at read time, with both inputs still on the table.
The practical shape is an append-only event log plus derived read models:
CREATE TABLE service_events (
event_id TEXT PRIMARY KEY,
work_order_id TEXT NOT NULL,
event_type TEXT NOT NULL,
device_id TEXT NOT NULL,
lamport BIGINT NOT NULL,
server_seq BIGSERIAL,
body JSONB NOT NULL,
recorded_at TIMESTAMPTZ NOT NULL, -- device wall clock, advisory
received_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE UNIQUE INDEX service_events_dedupe ON service_events (event_id);
Late arrivals stop being a problem. An event that shows up nine hours after the fact because the van was parked outside coverage all afternoon simply lands in the log and triggers a re-projection. There is no row to overwrite and no ordering to violate, because the log accepts insertions anywhere and the projection is a pure function over its contents.
The cost is real and I will not pretend otherwise. You now maintain projections, which can be wrong, can lag, and must be rebuildable. Querying gets more indirect. Developers who have never worked this way will fight it for a month. In exchange, you get a system where "what did the tech actually record" has an exact answer, and where two devices touching the same job do not silently destroy each other's work.
Not everything deserves this treatment. Mutable rows are fine for things only one authority ever changes — the customer's mailing address, edited by office staff on a connected machine, does not need an event log. Apply the pattern where field devices originate facts, and leave the rest alone.
Conflict Resolution, Honestly Compared
Assume you cannot avoid all conflicts. What are the actual options?
Last-write-wins. One version field, highest value takes the record. It is trivial to implement, and it is what you get by default when you do nothing. What it destroys is invisible: the losing write vanishes with no artifact. In a field context that means the office updates a work order's notes at 14:00 while the van, offline since noon, updates the same record from the job site. Whichever syncs later obliterates the other's work entirely — not the conflicting field, the whole row, including fields the loser was the only one to touch. LWW is defensible for genuinely single-writer data and for values where staleness is harmless. For a shared work order it is a data-loss generator with good marketing.
Per-field merge. Track a version per field rather than per record. Now the office's notes edit and the van's status update coexist, because they touched different columns. This handles the overwhelming majority of real-world "conflicts," which are not conflicts at all but concurrent edits to disjoint fields. The metadata cost is a version and origin per field, which is annoying but bounded. When two writers genuinely touch the same field you are back to needing a rule, but you have shrunk the problem to the cases that actually deserve thought.
Operational transformation. Transform incoming operations against ones already applied so intent survives reordering. It is the right tool for collaborative text — two people editing a note simultaneously, character by character. It is also notoriously difficult to get right, requires a central coordinator in most practical formulations, and demands transformation functions for every operation pair. For a field service app where the "collaborative editing" amounts to a free-text note that two people rarely touch at once, OT is enormous overkill. Do not build it.
CRDTs. Data types that converge by construction, no coordinator required.
- A G-Counter — grow-only, per-replica counters summed on read — fits monotonic tallies. Units of work completed across the whole van fleet, where each device only ever increments its own slot, converges correctly with no coordination at all.
- An LWW-Register gives you last-write-wins with an explicit, defensible tiebreak: a logical timestamp, and device ID as a deterministic tiebreaker so every replica picks the same winner. It is still last-write-wins, but it converges rather than depending on arrival order, which is a meaningful improvement over the accidental version.
- An OR-Set (observed-remove set) handles collections where add and remove race. Each add gets a unique tag; a remove only eliminates tags it actually observed. If one device adds "TPMS service required" to a job's flags while another removes it based on an older view, the add survives — which is usually what you want for a safety-relevant flag. The trade-off is tombstone growth; every removal leaves a marker, and unbounded tag accumulation is a real operational cost that you manage with periodic compaction under a coordinator.
CRDTs earn their keep for sets, counters, registers, and collaborative structures. They are libraries, not a philosophy, and using one does not require converting your entire model.
Where Automatic Convergence Is the Wrong Answer
This deserves its own section because it is where enthusiasm for CRDTs does the most damage.
A CRDT guarantees that all replicas reach the same state. It says nothing about whether that state is correct in a business sense. Convergence is a mathematical property. Correctness is a judgement about the world.
Take a concrete case. A tech in the field marks a job's tire disposal count as 4. Office staff, working from a customer's phone message, sets the same field to 2. These values converge under any CRDT you like — the LWW-Register picks one deterministically, both replicas agree, and everyone is happy except the person doing month-end reconciliation, because two humans disagreed about a physical fact and the system quietly picked a winner without telling anyone.
The right behaviour is to stop. Record both claims, mark the entity as contested, and route it to a person. Something like:
type FieldResolution<T> =
| { kind: 'converged'; value: T }
| { kind: 'contested'; claims: Claim<T>[]; requiresHuman: true };
const AUTO_MERGEABLE = new Set([
'internal_notes', 'photo_refs', 'inspection_flags', 'arrival_marker',
]);
function resolve<T>(field: string, claims: Claim<T>[]): FieldResolution<T> {
if (claims.length === 1) return { kind: 'converged', value: claims[0].value };
if (AUTO_MERGEABLE.has(field)) return { kind: 'converged', value: mergeCrdt(claims) };
return { kind: 'contested', claims, requiresHuman: true };
}
The allowlist is the design. You enumerate the fields where automatic convergence is acceptable and you default everything else to human review. That default is the opposite of what most sync libraries give you, and it is the correct one for anything with money, liability, or a physical count attached.
My rough test: if two people looking at the same conflict would need to talk to each other to settle it, the machine should not settle it silently. Quantities of parts consumed, whether work was authorized, what a customer agreed to, whether a tire was deemed repairable — these are disagreements between humans wearing a data costume. Converging them is not resolution, it is suppression. And a suppressed disagreement about whether a sidewall puncture was repairable is precisely the kind of thing that surfaces months later in a complaint, when the reasoning behind what makes a puncture fixable suddenly matters to someone with a lawyer.
Device Clocks Are Not Evidence
Every offline design eventually hits ordering, and every naive design reaches for a timestamp.
Device clocks are wrong in specific, repeatable ways. NTP sync fails when there is no network, which is exactly when the device is accumulating mutations. Cheap tablets drift measurably over a week. Users change the clock, sometimes to make a stubborn app behave. A device that has been powered off in a cold van overnight may come up with a nonsense clock before it syncs. And Alberta's spring-forward gives you a local hour that does not exist plus a fall-back hour that happens twice, so any local-time ordering has an annual guaranteed failure.
Rules that follow:
Store UTC and store the originating zone identifier separately — America/Edmonton, never a fixed offset, because the offset changes twice a year and a stored -07:00 is wrong for half the year. Never order by wall clock across devices. Keep the wall clock anyway, as advisory metadata, because it is genuinely useful for humans reading a record and for detecting a device whose clock has gone feral.
For ordering, use logical clocks. A Lamport clock is a single integer per replica, incremented on local events and advanced to max(local, received) + 1 on receipt. It gives you a total order consistent with causality: if A caused B, A's clock is lower. It cannot tell you whether two events were concurrent or merely unrelated, which is the limitation you accept for its simplicity.
class LamportClock:
def __init__(self, start: int = 0) -> None:
self._t = start
def tick(self) -> int:
self._t += 1
return self._t
def observe(self, remote_t: int) -> int:
self._t = max(self._t, remote_t) + 1
return self._t
A vector clock — one counter per replica, compared component-wise — does tell you about concurrency. If neither vector dominates the other, the events are genuinely concurrent and you have detected a real conflict rather than guessed at one. The price is size proportional to the replica count, which is fine for a dozen vans and unpleasant for a hundred thousand phones.
Then there is the server sequence, and for a hub-and-spoke topology it is the pragmatic winner. Devices sync through one backend. That backend assigns a monotonic sequence number on receipt, and that is the canonical order of record. Devices carry Lamport clocks so causality within a device's own stream is preserved, and the server sequence supplies the global ordering. You get a defensible, replayable order without vector clock overhead.
Ordering You Can Defend in a Dispute
Worth separating out, because the requirement is different from the engineering one.
Months after a job, someone asks a question with consequences attached: what tread depth was recorded, in what order did events happen, when did the customer approve the extra work. If your answer is "well, the timestamps say," and the timestamps came from a tablet whose clock nobody verified, you do not have a record. You have an assertion.
What makes an ordering defensible is that it is independently checkable and hard to alter after the fact. Three things get you most of the way there.
Keep both clocks. The server assigns received_at and server_seq on arrival; the device's claimed recorded_at is stored beside them, never overwriting. A large gap between them is not an error — it is the expected signature of a van that was out at an acreage all afternoon — but it is visible, and visible is what you need.
Make the log append-only in practice, not just in intent. Revoke UPDATE and DELETE on the event table for the application role. Corrections become new events that supersede prior ones, with a pointer to what they supersede and a stated reason. The original stays. This is the same instinct behind good paper records: you strike through and initial, you do not erase.
Chain the events per work order — each event stores a hash of the previous event's identifier and body. It is cheap, it is not a blockchain, and it means any after-the-fact edit to history breaks a verifiable chain. When someone questions a service record, you can demonstrate the sequence has not been rewritten, which is a meaningfully stronger position than trusting your own database.
The same instinct applies to any record where a measurement drives a decision. If a set of readings is what justified telling a customer their tires were near the end of their service life, the ordering and provenance of those readings is the substance of the conversation. Understanding how to read what the tire itself is telling you is the human half of that; the data model is the half that has to still be true next spring.
Two Weeks Behind on an Old Build
A van goes to a remote job. The tablet does not see a network for eleven days. Meanwhile the backend ships four releases. When the tablet reconnects it is running an old build and holding a fat stack of pending mutations written against an older schema.
If your API assumes client and server versions match, this is where everything breaks — and it breaks at the worst moment, with a queue full of real work nobody can reproduce.
Design against it directly. Every payload carries an explicit schemaVersion. Never infer it from build numbers or user agents; those lie, and they lie hardest during staged rollouts. Additive changes only within a major version: new fields must be optional with server-side defaults, and existing fields never change meaning. If a field's semantics must change, it becomes a new field. Renaming tread_mm to tread_depth_mm mid-flight is how you silently lose measurements from every device that has not updated.
The unknown-field question deserves a firm answer, and the answer differs by direction.
Server receiving unknown fields from a client: quarantine, never reject. A newer client sending a field the server does not recognize should not have its mutation refused — that mutation contains real work. Store the unrecognized keys in a sidecar JSONB column, apply what you understand, and emit a metric. Rejecting means throwing away a tech's afternoon because of a deployment ordering problem.
Client receiving unknown fields from the server: ignore, preserve, echo back. The classic bug is a client that reads a record, drops the fields it does not understand, and writes the whole object back — annihilating data written by newer clients. Preserve the unknown keys verbatim and include them in any subsequent write.
{
"schemaVersion": 7,
"entityType": "tread_reading",
"entityId": "8f21-...",
"known": { "position": "right_front", "valueMm": 5.5 },
"_unknown": { "gaugeSerial": "DG-4417", "ambientC": -21 },
"_originBuild": "van-app 3.2.1"
}
Two more rules. Deprecate on a clock the field can meet: a version stays supported for at least as long as your worst realistic offline window plus your slowest update cycle, and for tablets that update over an office connection during changeover season, that is months, not weeks. And expose a compatibility endpoint the app can consult on reconnect — minimum supported version, current version, whether a forced update is required before draining. A tablet that discovers it is too old should say so plainly and keep its queue intact rather than dumping it into a rejection loop.
Photos Must Not Block the Record
Job photos are the obvious "we'll add it later" feature that quietly wrecks a sync design.
A tech photographs a bulge in a sidewall, a wear pattern, a torque sequence, the odometer at an oil change. Each image is a few megabytes. A stop might produce a dozen. On a good connection that is a few seconds. In a parkade it is zero bytes for an hour, and on the drive back to town it is a long, interrupted transfer over a connection that will hand off between towers several times.
The rule: metadata records never block on bytes. They are separate channels with separate lifecycles.
The mutation log carries a photo reference — an identifier, a content hash, dimensions, capture metadata, and the local file path. That reference syncs in milliseconds along with everything else from the job. The image itself goes into a separate upload worker with its own queue, its own retry policy, and its own bandwidth policy. The server accepts a photo reference for content it does not yet hold and marks it pending, because a reference to bytes that have not arrived is a completely normal state.
CREATE TABLE photo_refs (
photo_id TEXT PRIMARY KEY,
work_order_id TEXT NOT NULL,
sha256 TEXT NOT NULL,
byte_size BIGINT NOT NULL,
captured_at_utc TIMESTAMPTZ NOT NULL,
upload_state TEXT NOT NULL, -- 'pending'|'uploading'|'stored'|'orphaned'
bytes_received BIGINT NOT NULL DEFAULT 0,
upload_session TEXT
);
Uploads must be resumable and chunked. Fixed chunks of a few hundred kilobytes, each acknowledged individually, with a session identifier that survives process restarts. A transfer interrupted at 80 percent resumes from 80 percent; a whole-file PUT that restarts from zero on every interruption will never complete on a marginal connection, and the worker will burn battery discovering that repeatedly.
Content-hash the file at capture time. It gives you free deduplication when the same image is somehow submitted twice, it lets the server verify integrity, and it makes the reference meaningful independent of transfer state.
Be deliberate about bandwidth. Wi-Fi at the yard or over a hotspot is not the same resource as cellular data on a rural highway, and battery in a cold van is a scarce input. A sane policy defers full-resolution transfers to Wi-Fi while pushing a small thumbnail immediately over cellular, so the office can see something is documented even if the full image lands hours later.
The orphan case needs a plan. Bytes uploaded whose metadata never arrives, and references whose bytes never come. Both happen. Sweep for references stuck pending beyond a threshold and surface them; sweep for uploaded blobs with no referencing record and expire them on a schedule. Neither should be discovered by a customer asking where the photo of their damaged rim went.
Pull Side: Cursors, Deltas, and Tombstones
Everything above is the push direction. The device also needs to learn what changed elsewhere — new jobs assigned, route changes, updated vehicle records.
The shape that works is a change feed with an opaque cursor. The server maintains a monotonic sequence over changes; the device stores its position and asks for everything after it.
GET /v1/changes?since=cursor_01J9F2M0Q7&limit=500
200 OK
{
"changes": [
{ "seq": 918442, "entity": "work_order", "id": "e2f1", "op": "upsert", "body": { } },
{ "seq": 918443, "entity": "work_order", "id": "a77c", "op": "tombstone",
"retiredAt": "2026-08-20T16:04:11Z", "reason": "customer_rescheduled" }
],
"nextCursor": "cursor_01J9F2M0RB",
"hasMore": true,
"backfillHorizon": "2026-05-22T00:00:00Z"
}
Make the cursor opaque. The moment a client parses it as a timestamp, you can never change the underlying implementation, and someone will eventually construct one by hand.
Deletes must be tombstones, always. A row that simply disappears from the server is invisible to a device that never asked about it specifically — the device will keep showing a job that no longer exists, and the tech will drive to it. A tombstone is an explicit change event carrying the identifier, the retirement time, and ideally a reason, because "why did this job vanish from my list" is a question with operational consequences when the answer is a customer who moved their slot. Tombstones need a retention floor longer than your maximum plausible offline window; prune them earlier and a long-disconnected device resurrects deleted records on reconnect.
Bound the backfill. A device that has been dark for months should not receive the entire history. Publish a horizon, and when a device's cursor falls behind it, respond with a reset instruction: wipe local server-derived state and take a fresh snapshot. Critically, that reset must not touch the device's own unsynced mutations. Those are the one thing on the tablet that exists nowhere else.
Scope the feed. A van in the northeast has no business receiving every work order in the region. Filter server-side by assignment and by service area, which keeps payloads small and is the same boundary you already reason about when deciding which parts of the region a van actually serves on a given day.
Detecting Divergence Nobody Reported
Sync systems fail quietly. That is their defining operational characteristic, and it is why reconciliation is not optional.
The failure that scares me is not the loud one. Loud failures — a queue erroring visibly, an upload stuck at 0 percent — get reported by the tech within the hour. The quiet failure is a device that believes it is fully synced while holding state the server does not have, or showing state the server has since changed. Everything looks green. The divergence surfaces weeks later as a work order with missing readings or a customer record that disagrees with itself.
Periodic checksum comparison catches it cheaply. The device computes a rolling hash over its records per entity type within a time window and sends the digest; the server computes the same and compares. Matching digests mean agreement without transferring anything. Mismatches trigger a narrowing search — split the window, compare halves, recurse until you have identified the specific records that differ. It is a Merkle comparison in spirit, and the point is that the common case costs a few hundred bytes.
def reconcile(device_digests, fetch_server_digest, drill):
"""Narrow to differing buckets; device_digests maps window -> hash."""
suspect = []
for window, local_hash in device_digests.items():
if fetch_server_digest(window) != local_hash:
suspect.append(window)
for window in suspect:
if window.span_hours <= 1:
drill(window) # enumerate and compare record by record
else:
reconcile(split(window), fetch_server_digest, drill)
Poison mutations need a defined path. A mutation that fails validation repeatedly must not retry forever, and it must not be dropped. Move it to a poison state after a bounded number of attempts, keep the payload and the last error, and surface it to a human who can decide. In this domain the humane resolution is usually not automated repair — it is showing the tech what they recorded and letting them re-enter it, because they were there and the server was not.
Watch queue shape, not just depth. A queue of forty mutations is unremarkable during a rural stop. The same forty mutations still present three days later is a broken system. Depth alone is a poor signal; oldest-pending-age is the honest one.
What Actually Deserves to Wake Someone
Alert on conditions that mean the system is losing data or is about to. Everything else is a dashboard.
Oldest pending mutation age, per device, is the primary signal. If a device's oldest unsynced mutation crosses a day, something is wrong that the tech has not noticed or has worked around. Two days is a fire.
Poison mutation count crossing zero deserves attention on the same day, because each one is a piece of work somebody did that the system has refused. Devices whose cursor has fallen behind the backfill horizon deserve a page before the tech opens the app and finds an empty schedule.
Silent devices — no sync attempt at all in an unexpectedly long stretch — are ambiguous but worth surfacing. The tablet may be broken, the van may be down, or the app may have wedged in a background state the OS never revives.
The metrics that inform without paging: reconciliation mismatch rate, drain latency distribution, photo upload backlog, schema version spread across the fleet. That last one is quietly the most useful during a rollout; a long tail of old versions tells you exactly how long your compatibility window really needs to be, as opposed to how long you wished it were.
One anti-pattern to name: alerting on individual request failures. In this environment failed requests are the normal state. A van driving out toward a rural job will fail every request for a stretch and that is the system working correctly. Alert on the outcome — work not landing — never on the attempt.
Testing With Deterministic Chaos
You cannot test this by hoping. The bugs live in interleavings a human will not think to try.
Build a simulator with a virtual clock. No wall-clock sleeps, no real sockets. A transport interface the tests control, so a test can pause delivery, duplicate a message, reorder two, deliver something after a nine-hour gap, or deliver a response to the server while dropping the acknowledgement on the return path — that last one being the exact shape that produces duplicate work orders.
def test_ack_lost_after_apply(sim):
dev = sim.device("van-02")
dev.record_event("tread_measured", position="left_rear", value_mm=4.0)
sim.partition(direction="response_only") # server applies, device hears nothing
sim.drain(dev)
sim.heal()
sim.drain(dev) # device retries with same key
assert sim.server.event_count(type="tread_measured") == 1
assert dev.pending_count() == 0
Seed every run and log the seed. When a randomized schedule finds a bug, the seed is the reproduction, and a bug you cannot reproduce in this domain is a bug you will not fix.
Property-based invariants are where the real value is. Generate random operation sequences across multiple devices with random partitions and skew, then assert properties that must hold regardless of schedule:
- No loss. Every mutation accepted locally eventually appears server-side, or sits in an explicit poison state. Never absent.
- No duplication. Applying the same mutation identifier any number of times produces one effect.
- Convergence. Once all partitions heal and all queues drain, every replica projects identical state for auto-mergeable fields.
- Causal consistency. If event B was recorded after A on the same device, no replica ever shows B without A.
- Monotonic cursors. A device's cursor never moves backward except through an explicit reset.
Run a reduced version in CI on every merge and a longer randomized soak nightly. Also test the boring paths that break in practice: an app upgrade with a non-empty queue, a database migration while mutations are pending, and a device restored from a backup taken before its last sync — which is a duplicate-identifier generator if the design is weak.
Finally, test on real hardware in real conditions. Simulators do not reproduce a tablet that has been in a -25 van since 6 a.m., a battery-saver mode that suspends background work, or an OS that kills the app during a long transfer. Drive into an actual parkade with an actual device. Winter changeover is a period where seasonal swap volume puts more stops on the schedule than any other time of year, and it is the load profile worth rehearsing against.
When You Should Not Build This
Most teams asking about offline-first should not build offline-first. I want to be blunt about that, because the pattern has enough intellectual appeal to survive its own cost-benefit analysis.
The cost is not the mutation log — that is a week. The cost is that every feature afterward carries an offline tax. New field? Decide its merge behaviour, its schema version, its conflict policy. New workflow? Work out what happens if it half-completes offline. Your test matrix multiplies. Debugging goes from reading one database to correlating several across time. Onboarding takes longer because the model is genuinely harder. That tax compounds for the life of the product.
Skip it when connectivity is genuinely reliable. An app used inside one building with decent Wi-Fi does not need this; it needs good retry behaviour and an honest error state.
Skip it when the work is read-mostly. If the field user is looking things up rather than originating records, cache aggressively, show the cache age plainly, and stop there. That is a caching feature and it is fine as one.
Skip it when brief offline windows are tolerable. If the tech can wait ninety seconds or step outside, an optimistic UI over a retry queue with a visible pending state gets you most of the value for a fraction of the complexity.
Skip it when conflicts are intrinsically human. If nearly every concurrent edit needs a person to adjudicate, automatic merging buys you nothing and the machinery is pure overhead. Build a good review queue instead.
And skip it when you cannot staff it. This is infrastructure, and it needs someone who understands it on the team over the long run. A brilliant sync layer built by a contractor who has moved on becomes the subsystem nobody will touch, which is worse than no sync layer, because now it fails and nobody knows why.
Build it when field workers originate durable records, connectivity failure is routine rather than exceptional, and the cost of not capturing at the moment of work is real. A mobile tire van meets all three. A tablet in a fixed bay does not, even though it is the same company doing the same kind of work.
A Staged Path If You Do Build It
If you have decided the case is real, sequence it so each stage is useful on its own.
Start with device-generated identifiers and client-minted idempotency keys, even on an otherwise conventional online app. It costs almost nothing, and it eliminates the duplicate-submission class of bugs immediately. If you do nothing else from this article, do this.
Next, put a durable local write log behind the UI with a visible pending indicator, still applying mutations against a normal API. Now the app survives short outages honestly, and you learn what your actual outage distribution looks like from real device data rather than guesswork.
Third, convert the entities where field devices are the origin of truth to an event model. Not everything — the readings, observations, and work records that come from the van. Leave customer profiles and catalog data as ordinary mutable rows.
Fourth, add the pull side properly: change feed, cursors, tombstones, bounded backfill. By this point you have real data about how far behind devices actually fall, which is what sets your retention floors.
Fifth, split binary attachments into their own resumable channel, with a policy about which network they are allowed to use.
Only then, if the evidence demands it, introduce CRDTs for the specific fields that need automatic convergence — and keep the contested-field path as the default for everything else.
At each stage, ship the observability with the mechanism rather than after it. A sync layer you cannot see into is a sync layer you cannot operate, and instrumentation added later is always thinner than instrumentation added alongside.
The Part That Matters More Than the Architecture
Every technique here exists to protect one thing: the accuracy of a record created by a person standing next to a vehicle.
When a tech measures 4 mm on a left rear and notes uneven wear across the axle, that observation is the input to a real decision — whether the tires finish the season, whether a rotation helps, whether the customer needs to plan for a set before winter rather than after the first storm. If the measurement is lost to a dropped request, duplicated into a confusing history, or silently overwritten by a stale edit from someone who was not there, the decision is made on bad information. The customer is the one who pays for that.
The domain is full of judgements that depend on trustworthy field data. Whether a puncture sits in the repairable zone or too close to the shoulder. What the load index on the sidewall permits for a loaded work truck. Whether an all-weather set suits someone's commute better than swapping twice a year, or whether the mileage they actually drive argues for dedicated seasonal rubber. Whether a vibration felt at highway speed points at wheel balance or at something well outside tire work. Those conversations only work when the history behind them is accurate, and for a mobile operation the history is assembled from records written in places with no signal.
Fleet work raises the stakes further. When a yard full of vehicles is on a recurring inspection cycle, the value is entirely in the trend — this unit's tread, tracked over time, against duty cycle. Lose or corrupt a fraction of those readings and the trend is fiction. Same for heavy units on a duty cycle, where a documented inspection history is part of how an operator plans downtime instead of absorbing it.
So the engineering discipline is not architectural fashion. It is the difference between a service record that means something and one that merely looks like it does. Design the data model for the parkade, the acreage, and the dead zone on the drive back — because that is where the work actually happens, and a system that only works with four bars is a system that does not work.
The details above are an engineering thought experiment about field data capture, written against the constraints of a Calgary mobile tire operation. Any figures mentioned are illustrative. Service information about tire work and oil changes is at calgaryrimandtire.ca.
