If you’ve ever run docker run hello-world, you’ve witnessed magic. A container spins up in milliseconds, isolated from your host, with its own filesystem, network, and process tree. But what actually happens when you hit Enter?
As DevOps engineers, we often treat Docker as a black box. We write YAML, push images, and hope for the best. But when things break—when a container won’t start, when disk space vanishes, or when networking fails—knowing the internals isn’t just "nice to have." It’s survival.
In this deep dive, we’ll peel back the layers of Docker. We’ll explore how Linux kernel primitives like OverlayFS and Namespaces power containers, decode the mysterious {} syntax in docker inspect, and learn how to troubleshoot like a pro.
1. The Image: It’s Not a File, It’s a Stack
Most people think a Docker image is a single binary. It’s not. An image is a stack of read-only layers.
How It Works
Every instruction in your Dockerfile (RUN, COPY, ADD) creates a new layer. These layers are stored in a content-addressable storage system (usually under /var/lib/docker/overlay2/).
- Union File System: Docker uses a Union File System (specifically OverlayFS) to merge these layers into a single view.
-
Content Addressability: Each layer is identified by a SHA256 hash. If two images share the same base (e.g.,
ubuntu:22.04), they share the same disk space. No duplication.
The JSON Truth
You can see this structure using docker image inspect. The output is a JSON object containing:
-
RootFS: A list of layer hashes. -
History: The build steps that created each layer.
docker image inspect nginx:latest | jq '.[0].RootFS.Layers'
Key Insight: Images are immutable. You never change an image; you only create new ones with additional layers on top.
2. The Container: Just a Process, But Isolated
A running container is simply a process on your host Linux kernel. But it’s a special process, wrapped in isolation primitives.
The Linux Kernel Primitives
Docker doesn’t invent virtualization; it leverages existing Linux features:
-
Namespaces: Provide isolation.
-
PID: The container sees only its own processes. -
NET: Its own network stack (IPs, ports). -
MNT: Its own filesystem view. -
UTS: Its own hostname.
-
- Control Groups (cgroups): Limit resource usage (CPU, Memory, I/O). This prevents one container from starving the host.
- OverlayFS Mount: The writable layer (the "container layer") is mounted over the read-only image layers.
Inspecting the Reality
When you run docker inspect <container_id>, you’re querying the Docker Daemon’s internal state database. The JSON output reveals the truth:
{
"State": {
"Status": "running",
"Pid": 12345,
"ExitCode": 0
},
"HostConfig": {
"Memory": 536870912,
"CpuShares": 1024
}
}
Key Insight:
docker runis essentially a wrapper aroundrunc(the OCI runtime), which sets up namespaces, cgroups, and mounts, then executes your entrypoint.
3. Networking: Virtual Wires and Bridges
How does a container talk to the world? Through Linux Network Namespaces and Virtual Ethernet (veth) pairs.
The Default Bridge
By default, Docker creates a Linux bridge (docker0) on the host.
- Each container gets a virtual interface (
eth0) inside its network namespace. - This
eth0is connected via a veth pair to thedocker0bridge. - NAT (iptables): Outbound traffic is masqueraded so it appears to come from the host IP.
Inspecting Networks
docker network inspect bridge
This returns JSON showing which containers are attached, their IPs, and MAC addresses.
Key Insight: Containers on the same bridge can communicate via IP. For name resolution, Docker runs an embedded DNS server at
127.0.0.11.
4. The Power of {{}}: Mastering docker inspect --format
One of Docker’s most powerful but underused features is the --format (or -f) flag. It uses Go’s text/template engine to parse JSON output directly in the CLI.
Why Use It?
docker inspect returns massive JSON blobs. Often, you just need one field. Parsing JSON with jq is great, but --format is built-in and faster for simple queries.
Syntax Rules
-
{{ }}: Delimiters for template expressions. -
.: Represents the current object. -
{{.Field}}: Accesses a key. -
{{.Parent.Child}}: Traverses nested objects. -
{{index .Array 0}}: Accesses array elements (Go templates don’t support[0]).
Real-World Examples
Get Container IP:
docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' my-container
List Environment Variables:
docker inspect -f '{{range .Config.Env}}{{.}}{{"\n"}}{{end}}' my-container
Check if Running:
docker inspect -f '{{if .State.Running}}✅ Up{{else}}❌ Down{{end}}' my-container
Format Mounts Like JSON:
docker inspect -f '{{range .Mounts}}
{
"Type": "{{.Type}}",
"Source": "{{.Source}}",
"Destination": "{{.Destination}}"
}
{{end}}' my-container
Pro Tip: Always wrap your template in single quotes (
'{{...}}') to prevent the shell from interpreting special characters.
5. Storage Deep Dive: OverlayFS & Copy-on-Write
Where do files live? How does Docker save space? The answer is OverlayFS.
The Layer Cake
Imagine your image has 3 layers:
- Base OS (
ubuntu) - App Dependencies (
nginx) - Your Code (
app)
When you start a container, Docker adds a 4th layer: The Writable Layer.
Copy-on-Write (CoW)
- Read: If a container reads a file from the image, it reads directly from the read-only layer. Fast.
-
Write: If the container modifies a file, Docker performs a copy-up:
- Copies the file from the read-only layer to the writable layer.
- Modifies the copy in the writable layer.
- The original remains untouched.
⚠️ Performance Warning: CoW copies the entire file, not just changed blocks. Modifying a 1GB log file triggers a 1GB copy. This is why you should use Volumes for databases and heavy I/O.
Directory Structure
/var/lib/docker/overlay2/
├── <layer-hash>/
│ ├── diff/ # Files unique to this layer
│ ├── lower # Link to parent layers
│ ├── merged/ # The unified view (mounted into container)
│ └── work/ # Internal workspace
6. Troubleshooting Like a Senior Engineer
When things go wrong, don’t guess. Inspect.
Step 1: Check Status
docker ps -a
Look at the STATUS and EXIT CODE.
-
0: Clean exit. -
137: OOM Killed (Out of Memory). -
125: Docker daemon error.
Step 2: Read Logs
Logs are stored in JSON files under /var/lib/docker/containers/<id>/<id>-json.log.
docker logs --tail 100 -f my-container
Step 3: Exec Into the Container
If the container is running but behaving strangely:
docker exec -it my-container /bin/sh
Now you can check files, network connectivity (curl, ping), and environment variables inside the namespace.
Step 4: Check Resources
docker stats
Real-time CPU, memory, and I/O usage. If a container is using 100% CPU, you’ll see it here.
Step 5: Disk Space
docker system df
Identify unused images, containers, and volumes. Clean up safely with:
docker system prune --dry-run # Always dry-run first!
Conclusion
Docker is not magic. It’s a clever orchestration of Linux kernel features:
- OverlayFS for efficient, layered filesystems.
- Namespaces for isolation.
- cgroups for resource control.
- Go Templates for powerful CLI inspection.
Understanding these internals transforms you from a Docker user into a Docker engineer. You stop guessing why a container failed and start knowing exactly where to look.
Next time you run docker run, remember: you’re not just starting an app. You’re creating a namespace, mounting a union filesystem, and isolating a process—all in milliseconds.
📚 Further Reading
Happy Containerizing! 🐳
