The Day We Accidentally Added Eight Minutes To Every Build
A platform team I worked with once spent weeks trying to optimize CI runners.
The symptoms looked obvious.
Builds were getting slower.
Developers were complaining.
Pipeline queues were growing.
Everyone assumed the issue was infrastructure.
Maybe the runners needed more CPU.
Maybe the shared runners were overloaded.
Maybe Docker caching wasn't working correctly.
After digging through build logs, the culprit turned out to be something far less exciting.
A single Dockerfile instruction had been moved.
That one change caused dependency installation to run on every build.
Nothing was wrong with Docker.
Nothing was wrong with the CI system.
We were invalidating the cache ourselves.
The frustrating part is that Docker was behaving exactly as designed.
The problem was that most engineers on the team didn't fully understand how Docker decides when a layer can be reused and when it must be rebuilt.
That's what this article is about.
We'll explore:
- Why Docker caching works the way it does
- What cache invalidation actually means
- Why Dockerfile ordering dramatically affects performance
- How BuildKit cache mounts take caching even further
- Practical patterns that can reduce build times from minutes to seconds
And perhaps most importantly:
How to stop fighting Docker and start working with it.
Docker Images Are Just Layered Filesystems
Most engineers understand Docker images conceptually.
Fewer understand how Docker actually builds them.
When Docker processes a Dockerfile, each instruction creates a layer.
Consider this simple example:
FROM node:22
WORKDIR /app
COPY package.json .
RUN npm install
COPY . .
RUN npm run build
Docker doesn't see this as one build.
It sees it as a sequence of independent layers.
Layer 1: Base Image
Layer 2: Working Directory
Layer 3: package.json
Layer 4: npm install
Layer 5: source code
Layer 6: application build
Each layer can potentially be reused.
That's where Docker's speed comes from.
The first build creates everything.
Subsequent builds reuse unchanged layers.
When this works correctly, builds become dramatically faster.
When it doesn't, Docker starts rebuilding large portions of the image.
Docker Doesn't Read Your Intentions
One important realization changed how I think about Dockerfiles.
Docker doesn't understand applications.
Docker doesn't understand dependencies.
Docker doesn't understand that changing a README shouldn't require reinstalling npm packages.
Docker only understands layers.
It compares instructions and inputs.
If something changes, that layer becomes invalid.
Everything after it must be rebuilt.
That's it.
No magic.
No application awareness.
Just layer dependency tracking.
The Cascade Effect Nobody Notices
Cache invalidation rarely affects a single layer.
It creates a chain reaction.
Imagine a Dockerfile like this:
FROM node:22
COPY . .
RUN npm install
RUN npm run build
Now suppose a developer changes one file:
src/api/users.ts
Only one source file changed.
However Docker sees:
COPY . .
as different.
That layer changes.
Which means:
RUN npm install
must run again.
Then:
RUN npm run build
must run again.
A tiny code change has now triggered a full dependency installation.
This is where many slow builds originate.
Cache Invalidation Is Usually Self-Inflicted
One of the most common complaints I hear is:
Docker cache seems unreliable.
In practice, Docker cache is remarkably predictable.
What engineers often interpret as cache problems are actually cache invalidation problems.
Docker follows a simple rule:
If a layer changes,
everything after it must be rebuilt.
Once you understand this rule, Docker behavior becomes easy to predict.
You stop guessing.
You start designing Dockerfiles intentionally.
Dockerfile Order Matters More Than You Think
A Dockerfile is not simply a list of commands.
It is a dependency graph expressed linearly.
The order directly affects cache efficiency.
Consider this version:
FROM node:22
COPY . .
RUN npm ci
RUN npm run build
This looks harmless.
It isn't.
Every source code modification invalidates:
- npm install
- build stage
The dependency layer never survives.
A Better Structure
Instead, separate dependency metadata from application code.
FROM node:22
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build
Now something interesting happens.
A code change affects:
COPY . .
and the build step.
But dependency installation remains cached.
The difference becomes massive in larger projects.
Why Dependency Layers Should Almost Never Change
Application code changes constantly.
Dependencies usually don't.
That's why dependency manifests should be isolated.
Examples:
Node.js
COPY package.json package-lock.json ./
RUN npm ci
Python
COPY requirements.txt .
RUN pip install -r requirements.txt
Rust
COPY Cargo.toml Cargo.lock ./
RUN cargo fetch
Go
COPY go.mod go.sum ./
RUN go mod download
This pattern exists across ecosystems.
The principle remains the same:
Cache expensive operations that change infrequently.
Traditional Docker Caching Has Limits
Even with proper layer ordering, there is another challenge.
Suppose this layer becomes invalid:
RUN npm ci
Docker must execute it again.
The cache layer is gone.
At first glance that seems unavoidable.
But modern Docker introduced a better solution.
BuildKit.
BuildKit Quietly Changed Docker Builds
Many teams are using BuildKit today without realizing it.
Modern Docker versions enable it by default.
BuildKit introduces several enhancements, but one feature stands out:
Cache mounts.
Unlike traditional layer caching, cache mounts preserve downloaded artifacts separately from image layers.
This changes everything.
Understanding Cache Mounts
Without BuildKit:
Layer invalidated
↓
Download packages again
↓
Install packages again
With BuildKit:
Layer invalidated
↓
Reuse package downloads
↓
Install packages
Notice the difference.
The installation still happens.
The expensive downloads don't.
BuildKit Cache Mounts In Practice
Node.js example:
# syntax=docker/dockerfile:1
FROM node:22
WORKDIR /app
COPY package*.json ./
RUN --mount=type=cache,target=/root/.npm \
npm ci
COPY . .
RUN npm run build
The cache directory persists between builds.
Future installations can reuse previously downloaded packages.
This becomes especially valuable in CI environments.
Python Example
RUN --mount=type=cache,target=/root/.cache/pip \
pip install -r requirements.txt
The same principle applies.
Packages are downloaded once and reused repeatedly.
Maven Example
RUN --mount=type=cache,target=/root/.m2 \
mvn package
Java builds often see dramatic improvements from this technique.
The Cost Savings Add Up
Engineers usually focus on speed.
The financial impact is often overlooked.
Faster builds mean:
- Less CI runner time
- Less network traffic
- Reduced artifact downloads
- Smaller cloud bills
In organizations running thousands of builds per day, these optimizations become surprisingly valuable.
Saving thirty seconds on one build sounds insignificant.
Saving thirty seconds on ten thousand builds per month is not.
Cache Strategy Is Really About Feedback Loops
At a technical level, we're discussing Docker layers.
At an organizational level, we're discussing developer productivity.
Every unnecessary rebuild delays feedback.
Every delayed feedback cycle slows delivery.
Good cache design isn't merely a Docker optimization.
It's an engineering productivity optimization.
The teams that deploy quickly rarely have magical infrastructure.
They simply eliminate waste everywhere they find it.
Dockerfile design is one of the easiest places to start.
Build Smarter, Not Harder
The biggest takeaway isn't a specific Docker command.
It's a mindset shift.
Docker isn't trying to make your builds slow.
Docker isn't randomly ignoring cache.
Docker follows deterministic rules.
Once you understand those rules, you can structure Dockerfiles to work with the caching system instead of against it.
Separate dependencies from source code.
Order instructions intentionally.
Use BuildKit cache mounts.
Preserve expensive operations.
And remember:
The fastest build isn't the one running on the largest CI runner.
It's the one that never had to do the work in the first place.
