Introduction
High availability features like managed instance groups (MIGs) and
load balancing are often treated as "set and forget" — configure them
once, and the platform handles the rest. In practice, understanding
when and how quickly these mechanisms respond to failure is
essential for setting realistic availability expectations. This post
walks through provisioning a three-tier architecture on Google Cloud
and deliberately introducing five distinct failure conditions to
observe and measure the platform's actual recovery behavior.
Solution overview
The architecture consists of a global HTTP load balancer distributing
traffic across a managed instance group, backed by a health check that
determines instance eligibility for traffic.
┌───────────────────────┐
Client ─▶ Global HTTP(S) LB │
└───────────┬───────────┘
│
┌────────────▼────────────┐
│ Backend Service │
│ (Health Check: /health) │
└────────────┬────────────┘
│
┌───────────────┼───────────────┐
▼ ▼
┌─────────┐ ┌─────────┐
│ VM 1 │ │ VM 2 │
└─────────┘ └─────────┘
Managed Instance Group (min:2, max:4)
Prerequisites
- Completion of Part 1, or equivalent familiarity with Compute Engine and firewall rules
- A load-testing tool such as
heyor Apache Bench installed locally or on a separate VM
Walkthrough
Step 1: Deploy the instance template and managed instance group
gcloud compute instance-templates create app-template \
--machine-type=e2-small \
--image-family=debian-12 \
--image-project=debian-cloud \
--tags=app-server \
--metadata-from-file=startup-script=startup.sh
gcloud compute instance-groups managed create app-mig \
--template=app-template --size=2 --zone=asia-southeast1-a
gcloud compute instance-groups managed set-autoscaling app-mig \
--zone=asia-southeast1-a \
--max-num-replicas=4 --min-num-replicas=2 \
--target-cpu-utilization=0.6 --cool-down-period=60
Step 2: Configure health checks and the load balancer
gcloud compute health-checks create http app-health-check \
--port=8080 --request-path=/health
gcloud compute firewall-rules create allow-health-check \
--allow=tcp:8080 \
--source-ranges=130.211.0.0/22,35.191.0.0/16 \
--target-tags=app-server
gcloud compute backend-services create app-backend \
--protocol=HTTP --port-name=http \
--health-checks=app-health-check --global
gcloud compute backend-services add-backend app-backend \
--instance-group=app-mig --instance-group-zone=asia-southeast1-a --global
gcloud compute url-maps create app-lb --default-service=app-backend
gcloud compute target-http-proxies create app-http-proxy --url-map=app-lb
gcloud compute forwarding-rules create app-http-rule \
--global --target-http-proxy=app-http-proxy --ports=80
Allow 5–10 minutes for the backend to transition to a HEALTHY state
before proceeding.
Failure scenario 1: Manual instance termination
gcloud compute instance-groups managed list-instances app-mig \
--zone=asia-southeast1-a
gcloud compute instances delete <INSTANCE_NAME> \
--zone=asia-southeast1-a --quiet
Observation: Measure the time between deletion and the appearance
of a replacement instance. If no replacement appears within several
minutes, verify whether autohealing is configured — MIG autoscaling
and autohealing are independent mechanisms, and autoscaling alone does
not guarantee replacement of a manually terminated instance outside of
its target size logic.
Failure scenario 2: Health check path blocked by firewall
gcloud compute firewall-rules delete allow-health-check --quiet
gcloud compute backend-services get-health app-backend --global
Diagnostic approach: From inside an affected VM, run
curl localhost:8080/health directly. A successful local response
combined with a failing get-health status isolates the fault to the
network path rather than the application.
Failure scenario 3: Load-induced autoscaling
hey -z 5m -c 200 http://<LB_IP>/
Observation: Track the interval between CPU utilization crossing
the 60% threshold and new instances becoming HEALTHY and receiving
traffic. Expect a multi-minute delay attributable to three sequential
stages: autoscaler metric evaluation, VM boot and startup script
execution, and health check confirmation before the load balancer
begins routing traffic.
Failure scenario 4: Misconfigured health check path
gcloud compute health-checks update http app-health-check \
--request-path=/healthz
Diagnostic approach: This produces the same UNHEALTHY status as
Scenario 2 but stems from an application-layer mismatch rather than a
network block — reinforcing that identical symptoms can have distinct
root causes, and log inspection (not assumption) should drive
remediation.
Failure scenario 5: Full instance group depletion
gcloud compute instance-groups managed resize app-mig \
--zone=asia-southeast1-a --size=0
Restore the health check path and resize back to 2, and measure total
recovery time from zero running instances to a healthy backend.
Clean up resources
gcloud compute forwarding-rules delete app-http-rule --global --quiet
gcloud compute target-http-proxies delete app-http-proxy --quiet
gcloud compute url-maps delete app-lb --quiet
gcloud compute backend-services delete app-backend --global --quiet
gcloud compute instance-groups managed delete app-mig \
--zone=asia-southeast1-a --quiet
gcloud compute instance-templates delete app-template --quiet
Conclusion
Each of these five scenarios produces a distinct combination of
symptoms and recovery timelines, despite relying on the same
underlying platform features. The key takeaway is that autoscaling,
autohealing, and load balancer health checking are separate,
composable mechanisms — understanding their individual behavior is
what separates confident incident response from guesswork.
In Part 3, we shift focus to IAM, auditing a project for
privilege escalation risks introduced through common configuration
shortcuts.
