09:41 PST. A support lead pastes a screenshot into the incident channel: her own screen, mid quick-switch into Acme Corp's account, showing three ticket subjects that mention Globex Inc, a different customer entirely. The message underneath is two words: "is this normal."
the setup
The internal tool support agents used to manage tickets across customer accounts was a multi-tenant React app on top of a GraphQL API, Apollo Client for data fetching. Tenant scoping happened entirely server-side: every request carried a JWT with a tenantId claim, and every resolver filtered its database queries by that claim. The GraphQL queries themselves never mentioned a tenant. A ticket list looked like this, the same query text for every account an agent ever loaded:
TicketsOverview.graphql
query TicketsOverview {
tickets {
id
subject
status
customer
}
}
That was a deliberate simplification. Plumbing a tenant argument through every query in the app would have meant threading it through dozens of components for no functional gain, since the server already refused to return data outside the caller's token. Nobody on the frontend team had reason to think of tenant identity as something the client needed to track for itself.
Six weeks earlier, product shipped Quick Switch: a dropdown that let an agent jump from one customer account to another without a full sign-out and page reload. The old flow called client.resetStore() on every account change, which cleared the Apollo cache and refetched every active query, a UX that took 300 to 500ms and showed a loading spinner over the whole page. Quick Switch skipped that on purpose, to make account switching feel instant:
useQuickSwitch.ts, before
async function switchTenant(tenantId: string) {
const { accessToken } = await api.post('/auth/switch-tenant', { tenantId });
setAccessToken(accessToken);
// resetStore() intentionally skipped here — it added a visible
// loading flash and product wanted the switch to feel instant.
}
Every query in the app used Apollo's default cache-first fetch policy. That was fine for a single-tenant session: read from cache if present, otherwise hit the network. Nobody had reasoned through what cache-first does the instant the same query, with the same variables, needs to mean something different because the identity behind it changed.
the scramble
First theory: a session bug on the auth service, a Redis-backed token cache serving a stale tenantId claim to the wrong pod. The on-call engineer pulled the auth service's logs for the support lead's account switch. The issued JWT was correct, Acme Corp's tenant ID, timestamped to the millisecond of the click. Ruled out within ten minutes.
Second theory: the GraphQL gateway was routing to a stale replica that hadn't caught up on a recent write, some kind of read-after-write lag. Someone checked the actual network response in the support lead's browser recording, the one piece of evidence that made this theory look promising at first. The GraphQL response body for TicketsOverview, once it arrived, contained exactly Acme Corp's tickets. Correct tenant, correct data, right there in the payload.
That was the detail that stalled the investigation for the better part of an hour: every request and every response, read individually, was correct. The bug wasn't in anything that crossed the network.
the hunt
The break came from reproducing it deliberately instead of waiting for another report. Someone throttled their local network to Slow 3G in dev tools, specifically to stretch out whatever window the bug lived in, then opened Apollo Client Devtools and watched the TicketsOverview cache entry while triggering Quick Switch.
The component re-rendered twice. The first render, milliseconds after the switch, showed the previous tenant's tickets, read straight from cache. The second render, 800ms to 1.5 seconds later depending on network conditions, showed the correct tenant's tickets, once the network response landed and overwrote the cache entry.
Apollo Client's default cache key is the query name plus its serialized variables. TicketsOverview had no variables. Every tenant, every session, every agent, all produced the identical cache key: TicketsOverview:{}. The cache had no idea two different customers' data had ever passed through that key, because from the cache's perspective, nothing about the query had changed. Tenant identity lived exclusively in an HTTP header the cache never looked at.
what cache-first actually did on switch
- Agent clicks "Switch to Acme Corp"
- New JWT (tenantId=acme) stored, old JWT (tenantId=globex) discarded
- TicketsOverview component re-renders
- Apollo checks cache for key TicketsOverview:{} -> HIT (Globex's data, still cached)
- cache-first policy: return the hit immediately, refetch in background
- UI paints Globex's tickets under Acme Corp's header, for 0.8-1.5s
- Network response for Acme's data arrives, overwrites cache, UI re-renders correctly
the find
Root cause: the Apollo Client cache had no concept of tenant identity, because tenant scoping had only ever been designed as a server-side concern. That was a correct design for the network layer, the server never returned the wrong tenant's data. It was an incomplete design for the client, because cache-first answers queries from a cache that has no way to know the caller's identity changed. Quick Switch removed the one step, resetStore(), that had been silently compensating for that gap by wiping the cache clean on every identity change. Once that step was gone, the gap was directly exposed to whoever happened to switch accounts fastest.
the fix
The immediate fix restored a cache clear on switch, but a narrower one than the old full resetStore(), which also triggers a refetch of every active query on the page, most of which weren't tenant-scoped UI at all:
useQuickSwitch.ts, after
async function switchTenant(tenantId: string) {
const { accessToken } = await api.post('/auth/switch-tenant', { tenantId });
await client.clearStore(); // empties the cache, no refetch of inactive queries
setAccessToken(accessToken);
// active queries now re-render against an empty cache and fetch fresh,
// instead of painting stale data first
}
clearStore() empties the cache without immediately refetching everything the way resetStore() does, so active components see a brief loading state rather than a full-page spinner, then fetch clean. That closed the immediate hole: an agent now sees a loading skeleton for a beat, never another customer's data.
The structural fix addressed the actual gap, that the cache had no tenant awareness at all, so the next feature that skips a manual clear doesn't reopen the same hole. Every tenant-scoped query now carries an explicit tenantId variable sourced from a reactive variable tied to the active session, and a type policy scopes the cache key by it:
cache.ts
export const activeTenantVar = makeVar(null);
export const cache = new InMemoryCache({
typePolicies: {
Query: {
fields: {
tickets: {
keyArgs: ['tenantId'],
},
},
},
},
});
TicketsOverview.graphql, after
query TicketsOverview($tenantId: ID!) {
tickets(tenantId: $tenantId) {
id
subject
status
customer
}
}
With tenantId in both the query variables and the type policy's keyArgs, Acme Corp's tickets and Globex's tickets now live under distinct cache keys, the same way two different pages of a paginated list already did. Even if a future feature forgets to clear the store on an identity change, the cache itself can no longer confuse one tenant's entry for another's; the identity is part of the key, not an invisible side channel.
the aftermath
1,900 Quick Switch actions recorded in the 30 days before the fix
46 Switches, across 12 agents, where session telemetry showed the stale render held for over 500ms
6 weeks Time between Quick Switch shipping and the first reported sighting
0 Stale cross-tenant renders recorded since the tenantId-scoped cache shipped
Because this was an internal tool, not customer-facing, no external data was exposed. It still landed as a corrective action under the company's SOC 2 tenant-isolation control, since that commitment covers who can see whose data regardless of which side of the product a screen sits on. A runtime assertion now runs on every tenant-scoped cache read in production: an Apollo Link compares the tenant embedded in a cached response against the tenant on the current session's JWT, and logs an alert if they ever disagree again. A synthetic test performs a scripted Quick Switch on a throttled connection in CI and fails the build if any tenant-scoped field renders data tagged with a different tenant ID than the active session, even for a single frame.
- A cache-first policy is only as safe as the assumption that the same query and variables always mean the same data. Tenant identity, or any other value carried outside the query itself, breaks that assumption the moment it changes without the cache knowing.
- Removing a slow safety step to make something feel instant is a real product tradeoff, not automatically a mistake, but it needs to be made with full knowledge of what the slow step was actually doing. Here it was quietly acting as the only thing scoping the cache by identity.
- A bug built entirely out of individually correct network requests and responses won't show up by inspecting the network tab alone. It has to be watched as a render sequence over time, which is why throttling the connection to stretch the window mattered more than any single log line.
- If a value determines which data a query is allowed to return, it belongs in the cache key, not only in a header. Otherwise the client is trusting the server's authorization to also double as the client's own cache boundary, and those are two different systems that happen to usually agree.
The server had been doing its job the entire six weeks, never once returning a ticket to a tenant that didn't own it. The leak was one layer up, in a cache that had no way of knowing two different customers had ever asked it the exact same question.
