Introduction
Disaster recovery plans are frequently validated on paper and rarely
tested end-to-end, which means the first real test often happens
during an actual incident. This post walks through provisioning
cross-region Cloud SQL replication and dual-region storage, then
deliberately removing the primary region's infrastructure to measure
actual Recovery Time Objective (RTO) and Recovery Point Objective
(RPO) against a documented estimate made in advance.
Solution overview
Region: asia-southeast1 (Primary) Region: asia-east1 (DR)
┌─────────────────────┐ ┌─────────────────────┐
│ Compute Engine VM │ │ │
│ Cloud SQL (Primary) │──replication▶│ Cloud SQL (Replica) │
└─────────────────────┘ └─────────────────────┘
│
▼
┌─────────────────────────────────┐
│ Cloud Storage (dual-region: ASIA1) │
└─────────────────────────────────┘
Prerequisites
- Two target regions selected for primary and DR roles
- Cloud DNS zone already configured for the application domain
Walkthrough
Step 1: Document your recovery estimate before building anything
Before provisioning any infrastructure, record a written estimate:
Estimated RTO: ____ minutes
Estimated RPO: ____ seconds/minutes
Anticipated recovery steps: 1. 2. 3.
Do not revise this document until after the drill — the comparison at
the end is the primary value of this exercise.
Step 2: Provision cross-region replication
gcloud sql instances create app-db-primary \
--database-version=MYSQL_8_0 --tier=db-g1-small \
--region=asia-southeast1 --backup --enable-bin-log
gcloud sql instances create app-db-replica-dr \
--master-instance-name=app-db-primary --region=asia-east1 \
--tier=db-g1-small
Step 3: Provision dual-region storage
gcloud storage buckets create gs://<PROJECT_ID>-dr-bucket --location=ASIA1
Step 4: Deploy the application and point DNS at the primary region
gcloud compute instances create app-vm-primary \
--zone=asia-southeast1-a --machine-type=e2-small \
--image-family=debian-12 --image-project=debian-cloud
gcloud dns record-sets create app.yourdomain-test.com. \
--zone=app-dr-zone --type=A --ttl=300 \
--rrdatas=<PRIMARY_VM_IP>
A 300-second TTL is used deliberately — higher TTLs common in default
registrar configurations directly extend recovery time by keeping
stale DNS resolutions cached at clients and resolvers.
Step 5: Simulate the disaster
Record the exact start time before proceeding — this is your RTO
measurement baseline:
date -u +"%Y-%m-%dT%H:%M:%SZ" > disaster-start-time.txt
gcloud compute instances delete app-vm-primary --zone=asia-southeast1-a --quiet
gcloud sql instances delete app-db-primary --quiet
Deleting the primary instance also orphans the read replica — a
realistic condition that must be handled as part of recovery.
Step 6: Execute recovery without referencing the original plan
gcloud sql instances promote-replica app-db-replica-dr
Deploy the application in the DR region:
gcloud compute instances create app-vm-dr \
--zone=asia-east1-a --machine-type=e2-small \
--image-family=debian-12 --image-project=debian-cloud
Expected friction point: connection strings hardcoded to the
original instance name will need manual updates once
app-db-replica-dr becomes the new writable primary. This is a direct
argument for externalizing connection configuration (via Secret
Manager or environment variables) rather than embedding it in
application code or images.
Update DNS to point at the DR region, and record the completion time:
gcloud dns record-sets update app.yourdomain-test.com. \
--zone=app-dr-zone --type=A --ttl=300 --rrdatas=<DR_VM_IP>
date -u +"%Y-%m-%dT%H:%M:%SZ" > recovery-end-time.txt
Verify DNS resolution independently (e.g., with dig) rather than
assuming propagation is instantaneous once the record is updated.
Step 7: Measure RPO
Cloud SQL read replicas replicate asynchronously; any writes to the
primary that had not yet replicated at the moment of deletion are
lost. To quantify this in a follow-up test run, write a record
immediately before deleting the primary and confirm whether it is
present in the promoted replica — this delta is your empirical RPO.
Comparing estimate to reality
| Metric | Documented estimate | Measured result | Delta |
|---|---|---|---|
| RTO | |||
| RPO |
Most teams running this drill for the first time find their initial
estimate omitted at least one dependency — commonly DNS propagation
delay or hardcoded connection configuration.
Clean up resources
gcloud compute instances delete app-vm-dr --zone=asia-east1-a --quiet
gcloud sql instances delete app-db-replica-dr --quiet
gcloud storage rm -r gs://<PROJECT_ID>-dr-bucket
gcloud dns managed-zones delete app-dr-zone --quiet
Conclusion
Replication and backups reduce data loss risk but do not eliminate
it — RPO is bounded by replication lag, not by the mere existence of a
replica. The most valuable output of this exercise isn't the
architecture itself, but the revised recovery runbook you write
immediately afterward, informed by what your original plan missed.
In Part 7, we shift from availability to cost, examining how
common infrastructure choices translate into avoidable cloud spend.
