Why HTTP 304 Not Modified Quietly Erased Our PWA State (And How We Fixed SingleFlight Cache Races with Mutation Epochs)

At 08:30 AM across a network of retail stores, dozens of employees opened our mobile Progressive Web App (PWA) to clock in for their morning shift.

A store associate tapped "Clock In". The spinner flickered for a brief moment, and a crisp green toast appeared:

"Clock-in Successful (HTTP 200 OK)". The PWA dashboard immediately transitioned to the active shift screen: "Currently On Duty".

Then, exactly one second later, something bizarre happened:

The UI jerked, the status flipped backward to "Not Clocked In", and the "Clock In" button reappeared as if nothing had ever taken place!

Naturally, the employee assumed the network had dropped their action and tapped the button a second time. This time, the server fired back an angry red error banner:

HTTP 400 Bad Request: "You are already clocked in for this shift!".

[The Employee's Dilemma]
- The screen insists: "You haven't clocked in yet. Tap to start!"
- The employee taps it: "Error: You are ALREADY clocked in!"

I'm Dương Phúc Khánh (also known internationally as Duong Phuc Khanh, and widely referred to as khanhdp or Kido, @khanhduong22). Over the past 7+ years—from building high-throughput financial trading systems at FPT Software (SBI Lab) to serving as Senior AI-Native Lead & Systems Architect at Tech Pulse—I've documented real-world production postmortems across 127 chapters of Kido Chronicles.

Today, we are dissecting this exact incident: A textbook collision between HTTP 304 Not Modified revalidation in client-side caching libraries, a SingleFlight Cache Race Condition, and the battle-tested architectural framework we engineered to eradicate it.

1. Autopsy #1: The HTTP 304 Not Modified Revalidation Trap

Why did a mutation that returned a successful HTTP 200 OK vanish from the client screen just 1,000 milliseconds later?

The Mechanics of SWR & React Query on Mobile PWAs

Modern frontend applications rely on "stale-while-revalidate" caching strategies (like swr or TanStack Query). When a user performs an action (e.g. clocking in), the client executes a mutation, updates its local optimistic UI state, and triggers a background revalidation of the primary dashboard data endpoint: GET /api/staff/home-data.

Here is the seemingly innocent sequence of events:

1. t = 0ms: The client sends POST /api/checkins.
2. t = 80ms: The backend server commits the shift to PostgreSQL, purges the Redis cache, and returns HTTP 200 OK with { status: "ACTIVE" }. React state updates instantly: isOnDuty = true.
3. t = 500ms: SWR initiates a background revalidation fetch for GET /api/staff/home-data.
4. THE LETHAL TRAP: The mobile browser engine (iOS WebKit / Android Chromium) automatically attaches conditional HTTP headers from its previous session: If-None-Match: "W/old-etag-pre-checkin".

sequenceDiagram
    autonumber
    actor Staff as Store Associate (Mobile PWA)
    participant SWR as SWR / React State
    participant Cache as Mobile Browser HTTP Cache
    participant API as Backend API (Hono / NestJS)
    participant DB as PostgreSQL Replica / Redis

    Staff->>API: 1. POST /api/checkins (Clock-In)
    API->>DB: Write to Primary DB + Invalidate Redis
    API-->>Staff: 200 OK { status: "ACTIVE" }
    Note over SWR: UI updates immediately: "On Duty" (0ms lag)

    par SWR Background Revalidation
        SWR->>Cache: GET /api/staff/home-data
        Cache->>API: GET with If-None-Match: "W/old-etag"
        Note over API,DB: Query hits Read Replica with 50ms lag!
        API-->>Cache: HTTP 304 Not Modified (0 bytes body)
        Note over Cache: Browser unpacks STALE JSON from disk cache!
        Cache-->>SWR: Returns stale pre-checkin data
        Note over SWR: SWR OVERWRITES REACT STATE WITH STALE DATA!
        SWR-->>Staff: UI rolls back to: "Not Clocked In" (Glitch!)
    end

The Culprit: Database Replication Lag & ETag Middleware

Behind the API gateway, read queries are routed to read replicas or cached application buffers:

• Replication Lag: Under burst morning load, the PostgreSQL read replica lagged behind the primary by just 30–50ms. WAL logs had not yet applied when the read query executed.
• ETag Calculation: The API server calculated an MD5/SHA-1 checksum of the response payload. Because the read replica returned the stale state, the newly computed hash was identical to the client's If-None-Match header.
• The server, complying with RFC 7234, returned HTTP 304 Not Modified without a response body.

The Backstab:

When the SWR fetcher received an HTTP 304, the browser intercepted it and served the stale cached response from disk. SWR treated this as fresh, verified truth and silently overwritten the live React state!

The freshly confirmed clock-in state was completely erased from the user's viewport by an obsolete cache entry.

2. Autopsy #2: The SingleFlight Cache Race (Slow Reads Punishing Fast Mutations)

Fixing ETag headers alone does not make you immune to concurrency bugs. Under the hood, we discovered an even nastier race condition: The SingleFlight Cache Race.

To prevent Cache Stampede / Thundering Herd problems when thousands of mobile clients request dashboard data simultaneously, high-performance backends implement the SingleFlight pattern (also known as request coalescing or in-flight promise sharing). If 10 requests arrive for cache:staff:123, only one triggers a database query, while the remaining 9 await the identical promise.

However, consider what happens when a fast write operation interleaves with a slow read operation:

A Timeline of the In-Flight Collision:

t = 0ms:    Request A (GET /api/staff/home-data) begins. 
            The DB is busy, so the complex multi-table SQL JOIN takes 150ms.
            Request A registers its in-flight promise for key "cache:staff:123".

t = 50ms:   User taps Clock In -> Request B (POST /api/checkins) arrives.
            Writing the new record to PostgreSQL takes only 20ms.

t = 70ms:   Request B purges the cache:
            await redis.del("cache:staff:123");
            Request B responds with 200 OK! The Redis cache is now EMPTY.

t = 150ms:  Request A finally finishes its 150ms SQL query!
            Unaware of the mutation that completed at t=70ms, Request A
            innocently populates the cache with its stale dataset:
            await redis.set("cache:staff:123", stalePreCheckinData, 'EX', 300);

THE DAMAGE: The cache, cleanly invalidated at 70ms, was RE-POLLUTED 
            with stale data at 150ms for the next 5 minutes!

sequenceDiagram
    autonumber
    participant Client
    participant SingleFlight as SingleFlight Manager
    participant Redis as Valkey / Redis
    participant DB as PostgreSQL Primary

    Note over Client,DB: t = 0ms: Request A (GET) starts slow query
    Client->>SingleFlight: GET /api/staff/home-data
    SingleFlight->>DB: Heavy SQL query (takes 150ms)

    Note over Client,DB: t = 50ms: Request B (POST Mutation) arrives
    Client->>DB: POST /api/checkins (takes 20ms)
    DB-->>Client: Mutation committed (t = 70ms)
    Client->>Redis: redis.del("cache:staff:123") (Cache purged!)

    Note over Client,DB: t = 150ms: Request A finishes stale query
    DB-->>SingleFlight: Returns old SQL payload
    SingleFlight->>Redis: redis.set("cache:staff:123", STALE_DATA, 'EX', 300)
    Note over Redis: STALE DATA RESURRECTED FOR 5 MINUTES!

A slow read that started before a mutation but finished after it completely undid the cache invalidation.

3. The 4-Layer Architectural Defense

To permanently solve this class of failures, Dương Phúc Khánh (khanhdp · Kido) designed a 4-Layer Defense-in-Depth Architecture:

┌─────────────────────────────────────────────────────────────┐
│ LAYER 1 (CLIENT): Confirmed Mutation Ledger (Local Storage) │
│ - Deterministic client reconciliation, immune to stale 304s │
├─────────────────────────────────────────────────────────────┤
│ LAYER 2 (SERVER): InFlight Mutation Epochs (resilient-cache)│
│ - Aborts stale cache writes whenever an epoch shift occurs  │
├─────────────────────────────────────────────────────────────┤
│ LAYER 3 (EDGE): Anti-304 Realtime Route Mandate             │
│ - Excludes dynamic write-sensitive routes from ETag hashing │
├─────────────────────────────────────────────────────────────┤
│ LAYER 4 (IDEMPOTENCY): 15-Second Double-Tap Soft Cushion    │
│ - Converts panic taps into idempotent 200 OK responses      │
└─────────────────────────────────────────────────────────────┘

Layer 1: Client Confirmed Mutation Ledger (@kt/spa-version-guard)

Never treat an HTTP response from a cellular network as infallible truth.

On the client side, we maintain a persistent Confirmed Mutation Ledger:

1. When a POST mutation succeeds, the client commits an immutable event to localStorage with a 12-hour expiration window.
2. Regardless of what the server or SWR returns on subsequent background revalidations, incoming data must pass through reconcileWithConfirmedLedger.
3. If the local ledger holds a verified clock-in timestamp newer than the server payload, the client overrides the stale server data in memory.
4. All dynamic queries enforce cache: 'no-store', and legacy Service Worker cache buckets are purged upon boot (caches.delete('legacy-api-cache')).

// Production slice from @kt/spa-version-guard
export interface ConfirmedRecord {
  id: string;
  type: 'CHECK_IN' | 'CHECK_OUT';
  confirmedAt: number;
  payload: Record<string, unknown>;
}

export class ConfirmedMutationLedger {
  private readonly storageKey = 'kt_confirmed_mutation_ledger_v1';
  private readonly ttlMs = 12 * 60 * 60 * 1000; // 12 hours

  recordMutation(type: ConfirmedRecord['type'], payload: Record<string, unknown>): void {
    const records = this.getActiveRecords();
    records.push({
      id: crypto.randomUUID(),
      type,
      confirmedAt: Date.now(),
      payload
    });
    localStorage.setItem(this.storageKey, JSON.stringify(records));
  }

  // Reconcile incoming server payloads against deterministic client truth
  reconcileHomeData<T extends { isCheckedIn: boolean; lastActionAt?: number }>(serverData: T): T {
    const activeRecords = this.getActiveRecords();
    const latestCheckin = activeRecords
      .filter(r => r.type === 'CHECK_IN')
      .sort((a, b) => b.confirmedAt - a.confirmedAt)[0];

    if (!latestCheckin) return serverData;

    const serverTimestamp = serverData.lastActionAt ?? 0;
    // If server reports false or has an older timestamp, client truth wins!
    if (!serverData.isCheckedIn && latestCheckin.confirmedAt > serverTimestamp) {
      return {
        ...serverData,
        isCheckedIn: true,
        lastActionAt: latestCheckin.confirmedAt,
        ...latestCheckin.payload
      };
    }

    return serverData;
  }

  private getActiveRecords(): ConfirmedRecord[] {
    try {
      const raw = localStorage.getItem(this.storageKey);
      if (!raw) return [];
      const parsed: ConfirmedRecord[] = JSON.parse(raw);
      const now = Date.now();
      return parsed.filter(item => now - item.confirmedAt < this.ttlMs);
    } catch {
      return [];
    }
  }
}

Layer 2: Server-Side InFlight Mutation Epochs (@kt/resilient-cache)

To kill the SingleFlight Cache Race, we augmented the coalescing engine with Mutation Epoch Counters:

1. An atomic counter, mutationEpoch, tracks all state alterations across the cache partition.
2. Whenever a mutation invalidates a key or pattern (e.g. staff:*), bumpMutationEpoch() increments the counter and detaches all active in-flight promises associated with that key.
3. Every read operation records a snapshot of startEpoch = currentEpoch before kicking off its database query.
4. Before persisting the result into Redis, it verifies: if (this.mutationEpoch !== startEpoch) return;. If a mutation intervened during the read, the write is rejected.

// Production architecture of InFlightManager in @kt/resilient-cache
export class InFlightManager {
  private inFlightPromises = new Map<string, Promise<unknown>>();
  private mutationEpoch = 0;

  bumpMutationEpoch(pattern?: string): number {
    this.mutationEpoch++;

    // Detach and evict in-flight promises matching the invalidated pattern
    if (pattern) {
      for (const key of this.inFlightPromises.keys()) {
        if (this.matchPattern(key, pattern)) {
          this.inFlightPromises.delete(key);
        }
      }
    } else {
      this.inFlightPromises.clear();
    }

    return this.mutationEpoch;
  }

  async dedupeWithEpoch<T>(
    key: string,
    fetcher: () => Promise<T>,
    onCommitToCache: (data: T) => Promise<void>
  ): Promise<T> {
    const startEpoch = this.mutationEpoch;

    if (this.inFlightPromises.has(key)) {
      return this.inFlightPromises.get(key) as Promise<T>;
    }

    const promise = (async () => {
      try {
        const result = await fetcher();

        // CRITICAL GUARD: Only commit to cache if no mutation occurred in between
        if (this.mutationEpoch === startEpoch) {
          await onCommitToCache(result);
        } else {
          console.warn(
            `[InFlightGuard] Discarded stale write for "${key}". Epoch shifted: ${startEpoch} -> ${this.mutationEpoch}`
          );
        }

        return result;
      } finally {
        this.inFlightPromises.delete(key);
      }
    })();

    this.inFlightPromises.set(key, promise);
    return promise;
  }

  private matchPattern(key: string, pattern: string): boolean {
    const regex = new RegExp('^' + pattern.replace(/\*/g, '.*') + '$');
    return regex.test(key);
  }
}

Layer 3: Anti-304 Realtime Route Mandate

For state-critical endpoints (/api/checkins/today, /api/checkins/status, /api/ip-status):

• We enforce strict non-cacheable headers that override intermediate proxies and CDNs:

  Cache-Control: no-store, no-cache, must-revalidate, proxy-revalidate
  Pragma: no-cache
  Expires: 0

• We explicitly exclude these routes from the backend's ETag generation middleware. Without an ETag response header, the browser will never issue an If-None-Match request, making an HTTP 304 structurally impossible.

Layer 4: 15-Second Idempotent Double-Tap Cushion

To safeguard user experience when an employee taps in panic:

• When a user repeats the exact same check-in action within 15 seconds of a successful request, the backend avoids throwing 400 Bad Request.
• Instead, it returns HTTP 200 OK with { deduplicated: true } and returns the previously generated shift record.
• Zero panic, zero confusing errors, and zero support tickets.

4. Production Telemetry & Results

Since shipping this 4-layer defense into production across our retail PWA infrastructure:

5. Architectural Takeaways

This postmortem reinforces three fundamental distributed systems principles:

1. HTTP 304 is a double-edged sword for Single Page Applications: HTTP caching is invaluable for static assets, but pairing automatic background revalidation with 304 Not Modified on mutable business state is an easy way to accidentally destroy client-side state.
2. Request Coalescing (SingleFlight) requires Mutation Epochs: Deduplicating concurrent database reads is critical for survival, but without epoch invalidation, slow reads will inevitably overwrite fast writes.
3. Embrace Client-Side Truth for Critical Actions: Under unpredictable mobile network conditions, client applications should maintain an authoritative local ledger of confirmed mutations rather than being at the mercy of lagging replicas.

About the Author (Tiểu sử tác giả): Dương Phúc Khánh (canonical handle: khanhdp, moniker: Kido) là Kỹ sư Kiến trúc Hệ thống (Senior AI-Native Lead & Systems Architect) tại Đà Nẵng, Việt Nam; cựu Tech Lead 6 năm tại FPT Software (SBI Lab), Quán quân Startup Runway 2018 (ĐH Kinh tế - ĐH Đà Nẵng), tác giả của Winter Arc, Agent Harness Kit và Kido Chronicles. Xem đầy đủ tại Tiểu sử Dương Phúc Khánh & Wikidata Q141718572.

About the Author & Ecosystem

Dương Phúc Khánh (also credited as Duong Phuc Khanh, khanhdp, or Kido) is a Senior AI-Native Lead & Systems Architect based in Da Nang, Vietnam, alumnus of Da Nang University of Economics (DUE), and creator of agent-harness-kit.

• 🌐 Official Architecture Hub & Case Studies: Dương Phúc Khánh (khanhdp)
• 🏛️ Biography & Engineering Timeline: Dương Phúc Khánh (khanhdp · Kido) (Tiểu sử)
• 🔥 Kido Chronicles (127 Real Production Postmortems): https://truyen.khanhdp.com
• 💻 Open-Source Multi-Agent Framework: github.com/khanhduong22/agent-harness-kit