Introduction
A deployment that reports success and a release that doesn't harm
users are not the same thing. This distinction matters most when a bug
doesn't manifest immediately — memory leaks, connection pool
exhaustion, and similar failure modes often pass initial health checks
before degrading. This post deploys a workload to Google Kubernetes
Engine with correctly configured liveness and readiness probes,
introduces a version with a 30-second delayed failure, and walks
through rollback and canary rollout as complementary mitigation
strategies.
Solution overview
┌─────────────┐ ┌──────────────────────┐
│ Ingress / │────▶│ Service: myapp │
│ Load Balancer │ └──────────┬───────────┘
└─────────────┘ │
┌─────────────┴─────────────┐
▼ ▼
┌───────────────────┐ ┌───────────────────┐
│ Deployment: stable │ │ Deployment: canary │
│ (9 replicas, v1) │ │ (1 replica, v2) │
└───────────────────┘ └───────────────────┘
Prerequisites
- A GKE Standard cluster with
kubectlconfigured - Artifact Registry for container image storage
Walkthrough
Step 1: Provision the cluster and registry
gcloud services enable container.googleapis.com artifactregistry.googleapis.com
gcloud artifacts repositories create app-repo \
--repository-format=docker --location=asia-southeast1
gcloud container clusters create app-cluster \
--zone=asia-southeast1-a --num-nodes=2 --machine-type=e2-medium
gcloud container clusters get-credentials app-cluster --zone=asia-southeast1-a
Step 2: Build an application with a controllable failure mode
from flask import Flask
import time, os
app = Flask(__name__)
START_TIME = time.time()
BUG_MODE = os.environ.get("BUG_MODE", "false") == "true"
@app.route("/health")
def health():
if BUG_MODE and (time.time() - START_TIME) > 30:
return "UNHEALTHY", 500
return "OK", 200
This mirrors real-world delayed failures more closely than a bug that
fails immediately at startup, which would be caught before any traffic
is served.
Step 3: Deploy with production-grade probes
readinessProbe:
httpGet: {path: /health, port: 8080}
periodSeconds: 5
failureThreshold: 2
livenessProbe:
httpGet: {path: /health, port: 8080}
periodSeconds: 10
failureThreshold: 3
kubectl apply -f deployment.yaml
kubectl get pods -w
Step 4: Roll out the faulty version and observe
kubectl apply -f deployment-v2-bugged.yaml
kubectl get pods -w
For the first ~30 seconds, pods report Running and Ready — this is
expected, and is precisely the window in which a naive smoke test
would report success.
Diagnosing probe behavior: After 30 seconds, inspect events:
kubectl describe pod <pod-name>
readinessProbe failures remove a pod from the Service's Endpoints
list without restarting the container; livenessProbe failures cause
the kubelet to kill and restart the container, visible as an
increasing RESTARTS count in kubectl get pods. Because this
example uses the same path for both probes, expect a repeating
restart cycle consistent with CrashLoopBackOff.
Step 5: Roll back
kubectl rollout undo deployment/myapp
kubectl rollout status deployment/myapp
Measure the elapsed time from issuing the rollback to full traffic
stability — this is your empirical Mean Time to Recovery (MTTR) for
this failure class.
Step 6: Implement canary rollout for the next release
Rather than replacing the entire fleet at once, split traffic across
two Deployments sharing a Service selector:
kubectl apply -f deployment-stable.yaml # 9 replicas, v1
kubectl apply -f deployment-canary.yaml # 1 replica, v2 (bug fixed)
for i in $(seq 1 50); do
curl -s http://<EXTERNAL_IP> | grep -o "v[12]-[a-z]*"
done | sort | uniq -c
Confirm the observed traffic split approximates the 9:1 replica ratio
before progressively shifting more replicas to the canary track.
Clean up resources
kubectl delete deployment myapp-stable myapp-canary --ignore-not-found
kubectl delete service myapp-service --ignore-not-found
gcloud container clusters delete app-cluster --zone=asia-southeast1-a --quiet
Conclusion
Readiness and liveness probes solve different problems and produce
different Kubernetes behaviors — conflating the two is a common source
of confusion when debugging rollout issues. Combined with a canary
rollout strategy, this pattern limits the blast radius of exactly the
kind of delayed-failure bug demonstrated here.
In Part 6, we validate disaster recovery assumptions by
deliberately deleting a region's infrastructure and measuring actual
recovery time against a documented plan.
