Debugging a Stubborn CrashLoopBackOff: Ephemeral Containers, kubectl debug, and Init Pitfalls

Your deployment rolled out. Thirty seconds later, kubectl get pods shows CrashLoopBackOff and a restart counter climbing. You reach for the usual reflex — kubectl exec -it pod/... -- sh — and get:

OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

Welcome to the modern container world: your image is distroless (or scratch, or a FROM scratch Go binary), there is no shell, no ps, no curl, no cat. And even if there were, the container is dead most of the time anyway — you can't exec into something that isn't running.

This article is a step-by-step method to diagnose that pod without ever rebuilding the image with "debug tools". We'll read exit codes properly, decode init container failures, attach ephemeral containers with kubectl debug, and clone a pod into a debuggable copy.

Context: why CrashLoopBackOff got harder

CrashLoopBackOff is not an error in itself. It's the kubelet telling you: "the container's main process exited, I restarted it, it exited again, so now I'm waiting before trying once more." The backoff starts at 10 seconds and doubles up to a 5-minute cap. That cap is why a crashing pod feels frozen after a few minutes — you get one short window of life every 300 seconds.

Two trends made this harder to debug:

1. Minimal images. Distroless, scratch, and Chainguard-style images ship a binary and little else. No package manager, no shell, often no writable filesystem. Great for the attack surface, terrible for kubectl exec.
2. Hardened Pod Security. On OpenShift, the restricted-v2 SCC forces a random UID, drops all capabilities, and blocks privilege escalation. Your old trick of "just run a privileged busybox" needs explicit authorization.

The good news: Kubernetes shipped ephemeral containers (stable since 1.25) precisely for this. You inject a throwaway container into the running pod — same network namespace, same volumes, optionally the same process namespace — without restarting anything.

Step 1: read the status before touching anything

Ninety percent of CrashLoopBackOff cases are solved here. Don't skip it.

kubectl -n shop get pod checkout-7d9f6c4b8-xk2lq -o wide
kubectl -n shop describe pod checkout-7d9f6c4b8-xk2lq

In describe, go straight to the Last State block. But the fastest, most reliable read is JSONPath, because describe truncates and reorders:

kubectl -n shop get pod checkout-7d9f6c4b8-xk2lq \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.exitCode}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.message}{"\n"}{end}'

And do the same for init containers — this is the single most forgotten command:

kubectl -n shop get pod checkout-7d9f6c4b8-xk2lq \
  -o jsonpath='{range .status.initContainerStatuses[*]}{.name}{"\t"}{.state}{"\n"}{end}'

Decoding exitCode correctly

exitCode is not noise. It's a precise signal:

Two important distinctions a beginner must internalize:

• 137 + reason: OOMKilled means the memory limit was hit. 137 without OOMKilled usually means a liveness probe killed the container after it ignored SIGTERM for terminationGracePeriodSeconds.
• exitCode 0 in CrashLoopBackOff is not a bug in Kubernetes. With restartPolicy: Always (the only option for Deployments), even a successful exit triggers a restart. If your container is a one-shot task, it belongs in a Job.

Logs of the previous instance

The container that failed is gone. Its logs are not in kubectl logs — they're one container generation back:

kubectl -n shop logs checkout-7d9f6c4b8-xk2lq -c checkout --previous
kubectl -n shop logs checkout-7d9f6c4b8-xk2lq --all-containers --previous --timestamps

If --previous returns nothing, the process died before writing anything — which itself points at exit 126/127 or a missing config file.

Make the crash message visible in the status

A trick almost nobody uses: terminationMessagePolicy. By default the kubelet only reads /dev/termination-log. Set it to FallbackToLogsOnError and the last lines of stderr are copied into lastState.terminated.message, visible in kubectl describe forever:

apiVersion: v1
kind: Pod
metadata:
  name: checkout
  namespace: shop
spec:
  containers:
    - name: checkout
      image: registry.example.com/shop/checkout:1.8.3
      terminationMessagePolicy: FallbackToLogsOnError
      terminationMessagePath: /dev/termination-log

Put this on every container in your base templates. It costs nothing.

Step 2: the init container pitfalls

If kubectl get pod shows Init:CrashLoopBackOff or Init:Error, your app container never even started. Init containers run sequentially, and each one must exit 0 before the next starts. A single failing init container restarts the whole pod.

Here's a realistic failing example — a distroless app behind a busybox init that waits for a database:

apiVersion: v1
kind: Pod
metadata:
  name: checkout
  namespace: shop
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  initContainers:
    - name: wait-for-db
      image: busybox:1.36
      command:
        - sh
        - -c
        - |
          until nc -z postgres.shop.svc.cluster.local 5432; do
            echo "waiting for postgres..." >&2
            sleep 2
          done
      securityContext:
        allowPrivilegeEscalation: false
        runAsUser: 65532
        capabilities:
          drop: ["ALL"]
  containers:
    - name: checkout
      image: gcr.io/distroless/static-debian12:nonroot
      args: ["--config=/etc/checkout/config.yaml"]
      terminationMessagePolicy: FallbackToLogsOnError
      volumeMounts:
        - name: config
          mountPath: /etc/checkout
          readOnly: true
      securityContext:
        allowPrivilegeEscalation: false
        runAsUser: 65532
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]
  volumes:
    - name: config
      configMap:
        name: checkout-config

Classic traps:

• Infinite wait, not a crash. Init:0/1 that never progresses isn't CrashLoopBackOff — it's a DNS or NetworkPolicy issue. Check with kubectl logs pod/checkout -c wait-for-db.
• The init container has a shell, the app doesn't. You can debug the init phase easily, which lulls you into thinking the problem is in the app.
• Sidecars as init containers (restartPolicy: Always on an init container, Kubernetes 1.29+) behave differently: they're not expected to exit, and the next init container starts as soon as they're Started. Mixing a classic init and a native sidecar in the wrong order is a common source of hangs.
• Init container logs disappear on restart. Use --previous on init containers too: kubectl logs pod/checkout -c wait-for-db --previous.

Step 3: attach an ephemeral container with kubectl debug

The pod still has a running sandbox (pause container, network namespace, volumes) even while the app container is dead. That's where you inject your toolbox:

kubectl -n shop debug -it checkout-7d9f6c4b8-xk2lq \
  --image=nicolaka/netshoot:v0.13 \
  --target=checkout \
  --profile=general \
  -- bash

What each flag does:

• --image — your debug toolbox. busybox:1.36, nicolaka/netshoot, or an internal image with curl, dig, ss, strace.
• --target=checkout — share the process namespace of that specific container. Without it you can't see or inspect the app process.
• --profile — general, baseline, restricted, netadmin, sysadmin. Always pass it explicitly; the implicit default has changed across versions. Use restricted on hardened namespaces, netadmin only when you need NET_ADMIN / NET_RAW for tcpdump.

Once inside, you are in the same network namespace as the pod:

# Is the config file actually mounted and readable?
ls -l /proc/1/root/etc/checkout/ 2>/dev/null

# Does the service resolve from inside the pod?
nslookup postgres.shop.svc.cluster.local
nc -vz postgres.shop.svc.cluster.local 5432

# Who is listening in this pod's netns?
ss -lntp

# What environment did the app actually get?
cat /proc/1/environ | tr '\0' '\n'

The /proc/<pid>/root trick is the key to distroless debugging: with --target and a shared PID namespace, you can read the other container's filesystem through /proc, even though your debug container has its own image. Find the PID with ps aux first — it won't always be 1.

Limits you must know

• Ephemeral containers cannot be removed. Once added, they stay in the pod spec until the pod is deleted. Debug, then delete the pod.
• They cannot define resources, ports, probes, or lifecycle. They have no resource requests — mind your node pressure.
• They need the pods/ephemeralcontainers subresource RBAC verb (see security section).
• If the pod has no sandbox at all — Pending, ImagePullBackOff, CreateContainerConfigError — ephemeral containers won't help. That's a scheduling/image/config problem; kubectl describe events are your answer.

Step 4: clone the pod when you need to change the command

Sometimes the container dies in 200 ms and you need it alive to inspect it. You can't change the command of a running pod — but you can copy it:

kubectl -n shop debug checkout-7d9f6c4b8-xk2lq \
  --copy-to=checkout-debug \
  --container=checkout \
  --keep-init-containers=false \
  -it -- sleep infinity

This creates a new pod checkout-debug with the same volumes, env, and service account, but the checkout container's command replaced by sleep infinity. Note: if the image is distroless, sleep doesn't exist — combine with --set-image:

kubectl -n shop debug checkout-7d9f6c4b8-xk2lq \
  --copy-to=checkout-debug \
  --set-image=checkout=busybox:1.36 \
  --keep-init-containers=false \
  -it -- sh

Now you have a shell with the exact same mounted ConfigMaps, Secrets, and ServiceAccount token. Validate the hypothesis:

cat /etc/checkout/config.yaml
ls -ln /etc/checkout         # UID/GID mismatch with runAsUser?
env | sort
cat /var/run/secrets/kubernetes.io/serviceaccount/namespace

Flags that matter on --copy-to:

• --keep-labels=false (the default) strips labels so the copy does not receive Service traffic. Never set it to true in production.
• --keep-init-containers=true is the default — set it to false when the init container is exactly what's failing, otherwise your copy crashes the same way.
• --keep-liveness, --keep-readiness, --keep-startup default to false: probes are removed, so nothing kills your debug pod.
• --share-processes makes all containers in the copy share the PID namespace.

Delete the copy when you're done: kubectl -n shop delete pod checkout-debug.

OpenShift specifics

oc debug predates kubectl debug and does the copy-and-neutralize dance for you:

# Debug a copy of the pod, probes removed, command replaced by a shell
oc -n shop debug pod/checkout-7d9f6c4b8-xk2lq --image=registry.access.redhat.com/ubi9/ubi-minimal

# Debug from a workload controller directly
oc -n shop debug deployment/checkout --image=registry.access.redhat.com/ubi9/ubi-minimal -- /bin/bash

# Keep the same UID/SCC context instead of asking for root
oc -n shop debug deployment/checkout --as-user=1000650000

If the problem looks node-level (disk pressure, CNI, image GC):

oc debug node/worker-02
# then, inside:
chroot /host
crictl ps -a | head
journalctl -u kubelet --since "10 min ago" | tail -50

The kubectl equivalent, with the node filesystem under /host:

kubectl debug node/worker-02 -it --image=busybox:1.36 --profile=sysadmin

Security best practices

Debugging tooling is a privilege-escalation vector. Treat it like one.

1. Scope RBAC to the pods/ephemeralcontainers subresource. Creating an ephemeral container means running arbitrary code inside an existing pod, with that pod's ServiceAccount token. Don't bundle it into a generic edit role.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-debugger
  namespace: shop
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods/log", "pods/exec"]
    verbs: ["get", "create"]
  - apiGroups: [""]
    resources: ["pods/ephemeralcontainers"]
    verbs: ["get", "patch", "update"]

Bind it to an on-call group, not to every developer, and prefer a time-bound binding you remove after the incident.

2. Never default to --profile=sysadmin. It grants privileged: true and host namespaces. Start with restricted, escalate to netadmin only for packet capture, and to sysadmin only for node-level work. On OpenShift, needing sysadmin means you need an SCC exception — make it explicit and auditable.

3. Pin and curate your debug images. nicolaka/netshoot:v0.13, not :latest. Mirror it into your internal registry, scan it, and allow it in your admission policy. A debug image with network tools running in a pod with a mounted ServiceAccount token is a lateral-movement toolkit.

4. Remember that a copied pod inherits the Secrets. --copy-to mounts the same Secrets and SA token. Anyone with shell access in the copy can read them. Delete the copy immediately, and never leave it running overnight "just in case".

5. Keep labels off the copies. The default --keep-labels=false prevents the debug pod from being picked up by a Service and serving real traffic. Verify with kubectl get endpoints -n shop before and after.

6. Audit it. Ephemeral container creation appears in the API audit log as a patch on pods/ephemeralcontainers. Add an alert on it in production namespaces — it's a low-volume, high-signal event.

7. Don't "fix" the problem by shipping a shell. Adding busybox to your production image to make debugging easier undoes the entire point of distroless. The whole workflow above exists so you don't have to.

Conclusion: the checklist

When a pod loops and you have no shell, run these in order:

1. describe + JSONPath on lastState.terminated — get the exitCode and reason. 127 = wrong command, 137 + OOMKilled = memory limit, 143 = something killed it, 0 = wrong workload type.
2. kubectl logs --previous, on the app container and every init container. Enable terminationMessagePolicy: FallbackToLogsOnError everywhere so the message survives in the status.
3. Check initContainerStatuses before blaming the app. Init: prefixes in the pod status mean the app never ran.
4. kubectl debug --target=<container> --profile=general for live inspection: DNS, connectivity, mounted files via /proc/<pid>/root, real environment via /proc/<pid>/environ.
5. kubectl debug --copy-to --set-image --keep-init-containers=false when you need the container to stay alive with a different command.
6. kubectl debug node/<node> --profile=sysadmin or oc debug node/... only when the evidence points below the pod.

And the mental model to keep: an exit code tells you how it died, the logs tell you why, and the ephemeral container tells you what the environment actually looked like — which, in nine cases out of ten, is different from what your manifest led you to believe.