Caching: Why Faster Reads Create Consistency Problems

A dashboard request can look harmless.

The problem appears when hundreds of users ask for the same expensive data at the same time.

In Software Engineering Lab 04, the scenario is a workshop dashboard. Without caching, every request runs six database queries with joins and aggregations.

500 concurrent users
        ↓
Dashboard request
        ↓
6 queries + join/aggregation per request
        ↓
3000 total DB queries

The first instinct is easy:

put Redis in front of the database

That reduces repeated work.

But it also creates a new set of problems.

How stale may the cached value be?
When should it be invalidated?
What happens when the key expires under load?
What happens if Redis is unavailable?
Can the cache return data from the wrong tenant?

That is the part I wanted to explore in Lab 04.

The main mental model is not:

slow query → add cache

It is closer to:

reduce repeated work
        ↓
accept a consistency boundary
        ↓
design expiration, invalidation, failure, and concurrency behavior

PostgreSQL is still the source of truth

The lab uses two storage layers:

Layer	Technology	Role
Primary	PostgreSQL	Durable, persistent, authoritative storage that can rebuild the cache
Cache	Redis	Derived data, TTL-bound, rebuilt on demand

The important decision is ownership of correctness.

In this lab, PostgreSQL remains authoritative for business data. Redis stores derived data that can disappear, expire, or be rebuilt.

That means cache correctness has to work even when the cache is empty.

The cache is an optimization layer, not the only copy of the business state.

The read path: Cache Aside

Lab 04 uses Cache Aside for reads.

GET cache
   ↓
hit? ───── yes ───→ return cached value
   ↓ no
query PostgreSQL
   ↓
populate Redis
   ↓
return value

A cache miss is not an application failure.

It means the application has to rebuild the value from the authoritative source.

The application explicitly knows about both storage layers:

Redis
PostgreSQL

That control is useful when Redis is unavailable because the application can still attempt to read from PostgreSQL, as long as the database and fallback capacity can handle the traffic.

But Cache Aside also means the application now owns cache freshness.

That is where the interesting failures start.

TTL is really a staleness decision

The useful question is not simply:

Does this data change?

The lab asks:

How long can stale data be accepted?

Examples recorded in the lab:

Data	Max staleness	Reasonable?
Dashboard statistics	30s–2min	Yes, operational metrics
Stock display	1–5s	Yes, UI/UX only
Wallet balance	0s	No, audit risk

The difference matters.

A dashboard metric can tolerate a freshness window that would be unacceptable for a transactional balance.

So TTL is not just an expiry configuration. In this implementation, it represents an accepted freshness window and also acts as a recovery path when stale cache survives longer than expected.

Invalidating before commit can reintroduce stale data

Consider this write flow:

DELETE cache
↓
update database
↓
COMMIT

It looks reasonable. Remove the old cache first, then write the new database value.

The race appears when a reader enters between those operations.

Lab 04 describes this timeline:

T1 Writer: DELETE cache
T2 Reader: cache MISS → reads old DB value
T3 Reader: SET old value into cache
T4 Writer: DB COMMIT new value

Final state:

Database = new value
Cache    = old value

This is stale cache, not data loss. The authoritative business data in PostgreSQL is still correct.

A safer order is:

DB COMMIT
↓
DELETE cache

Now a reader that misses after the delete can rebuild from the committed database value.

But even this is not strong consistency.

Another interleaving still exists:

T1 Reader: cache MISS
T2 Reader: reads old DB value
T3 Writer: DB COMMIT new value
T4 Writer: DELETE cache
T5 Reader: SET old DB result into cache

Final state again:

Database = new value
Cache    = old value

So the conclusion is narrower:

COMMIT → DELETE

is safer than:

DELETE → COMMIT

but Cache Aside is still an eventually consistent optimization in this lab.

TTL remains useful as a safety net for the residual stale window.

Updating Redis after a database write is not atomic either

The lab also uses an application-managed update-on-write flow:

DB update
↓
DB COMMIT succeeds
↓
best-effort Redis SET
↓
return success

The database returns the authoritative value, then the application tries to update the cache.

The problem is the boundary between PostgreSQL and Redis.

These are separate systems, so this sequence is not atomic:

DB COMMIT
↓
Redis SET

A process can fail between them:

DB COMMIT succeeds
↓
process crashes
↓
Redis SET never happens
↓
old cache remains

Concurrent writers create another race:

Writer A commits value A
Writer B commits value B
Writer B SET cache = B
Writer A performs a late SET cache = A

Final state:

Database = B
Cache    = A

This was an important distinction for me:

Cache Aside
→ read strategy

Invalidate-on-write / update-on-write
→ write strategy

They solve different parts of the cache lifecycle.

Neither turns PostgreSQL and Redis into one atomic system.

One miss is normal. One thousand simultaneous misses are not

Now consider a popular key reaching expiration.

cache expires
      ↓
1000 concurrent requests arrive
      ↓
1000 cache misses
      ↓
1000 parallel DB queries
      ↓
database overload / crash

This is the cache stampede scenario used in Lab 04.

The cache successfully removes repeated database work while the entry exists, but expiration can suddenly send that work back to the database at the same time.

For duplicate rebuilds inside one process, the lab uses golang.org/x/sync/singleflight.

The flow includes a second cache check:

initial cache GET
      ↓
miss
      ↓
singleflight.Do
      ↓
check cache again
      ↓
query DB
      ↓
populate cache
      ↓
share result

The second check matters because another caller may already have populated the cache between the first miss and the point where this caller becomes the rebuild leader.

Singleflight stops at the process boundary

Singleflight coordinates callers inside one process.

A multi-instance deployment needs a different coordination boundary.

Lab 04 implements a distributed-lock primitive:

WithLock() = try-once lock primitive

The conceptual cache regeneration flow is:

cache GET
   ↓ miss
acquire distributed lock
   ↓
check cache again
   ↓
query DB
   ↓
populate cache
   ↓
safe release

The lock requirements in the lab are explicit:

unique token or owner;
TTL to avoid a permanent deadlock;
atomic compare-and-delete when releasing;
one holder must not remove another holder's lock.

But even this has a boundary.

If regeneration takes longer than the lock TTL:

Instance A acquires lock
↓
lock expires
↓
Instance B acquires a new lock
↓
Instance A is still rebuilding
↓
duplicate rebuild can run

The lock in Lab 04 reduces duplicate cache regeneration. It is not used as a correctness primitive for business transactions.

Expiration can also be spread out

The lab adds TTL jitter so many keys do not expire at nearly the same moment.

Example:

60s + random 0–15s

The implementation produces values in:

[base, base + maxJitter)

The TTL never goes below base, and the upper bound is exclusive.

The lab also discusses background refresh: refresh the cached value before expiry while clients continue receiving the existing cache value.

Both techniques target expiration behavior rather than changing the source of truth.

Cache keys are part of the data boundary

The canonical key format in the lab is:

{app}:{tenant}:{branch}:{resource}:{dimension}

Example:

cmms:tenant:42:branch:7:dashboard:2026-09-01

The rule is simple:

Every input that changes the result belongs in the cache key.

For the dashboard example, that includes tenant, branch, and business date.

This is also a security boundary in a multi-tenant system.

A sensitive cached value must include tenant scope. Missing isolation can expose data under the wrong tenant context.

Key design also affects reuse.

If 10,000 users read the same branch dashboard, this shape:

cache:tenant:42:user:{user_id}:dashboard

creates high-cardinality entries for data that is actually shared.

The lab contrasts it with:

cache:tenant:42:branch:{branch_id}:dashboard

One branch-scoped value can be reused by those readers.

The trade-off is not "specific keys are bad." The point is that key dimensions should match the result boundary.

Redis failure changes where the traffic goes

Cache Aside allows database fallback when Redis fails, but only while the authoritative dependency and fallback capacity remain available.

The traffic pattern becomes:

traffic previously absorbed by Redis
      ↓
directly reaches PostgreSQL
      ↓
load spike / cache-failure amplification

So graceful degradation does not mean the outage has no effect.

It means the main function may continue with degraded performance while PostgreSQL can still handle the fallback traffic.

The lab also separates:

cache_miss

from:

cache_error

because an expected miss and an unavailable cache backend are different operational events.

Hit ratio alone does not tell me whether the cache is worth it

Lab 04 evaluates cache value together with:

avoided query cost
cache latency
memory cost
invalidation complexity
failure amplification

The README gives this comparison:

30% hit ratio for a 100ms operation
may still be valuable

99% hit ratio for a 0.1ms operation
may not justify the complexity

The useful signal is not the percentage alone.

The cost of the work being avoided matters.

Cache comes after understanding the database work

The diagnostic order in the lab is:

measure endpoint
↓
check N+1 queries
↓
inspect execution plan
↓
add / optimize indexes
↓
reduce selected columns
↓
optimize joins / subqueries
↓
evaluate caching if the workload needs it

This prevents cache from becoming a way to hide an unexplained query problem.

For a very cheap query and some workloads, the extra cache hop and operational complexity may not provide meaningful end-to-end benefit.

The mental model I take from this lab

The naive model is:

query is expensive
→ add Redis
→ problem solved

Lab 04 forces a longer chain of questions:

What is the source of truth?
↓
How stale may this value be?
↓
What dimensions belong in the key?
↓
How is the value invalidated after writes?
↓
What happens during concurrent misses?
↓
What happens when Redis fails?
↓
Is the avoided work worth the added complexity?

That is the main lesson from the experiment.

Caching can reduce repeated work in a read-heavy workload.

But the moment a second storage layer is introduced, freshness, invalidation, concurrency, failure behavior, and isolation become part of the design.

Running the lab

From the repository root:

docker compose up -d redis

make lab-04-test
make lab-04-test-race
make lab-04-vet
make lab-04-demo
make lab-04-integration

The demo scenarios can also be run directly:

cd labs/04-caching

go run ./cmd/demo -scenario=without-cache
go run ./cmd/demo -scenario=cache-aside
go run ./cmd/demo -scenario=stampede-unprotected
go run ./cmd/demo -scenario=stampede-protected
go run ./cmd/demo -scenario=write-through