I Built a Multi-Region Pilot Light on AWS. The Diagram Was the Easy Part.

Most disaster recovery write-ups stop at the diagram.

You get two boxes, an arrow labelled "replicate", and an RTO that was never measured. I wanted the rest of the story: real Terraform, a failover you can time, and a written answer for every trade-off including the ones that only look smart at demo scale.

So I built a notes API across two AWS regions in the pilot light pattern, then reviewed the whole thing against the Well-Architected Framework.

Repo: aws-multi-region-pilot-light

The application is small on purpose. The value is the architecture.

The pattern in one paragraph

eu-west-1 (Ireland) serves all traffic. eu-west-3 (Paris) continuously receives every database write and every S3 object, but runs zero application instances.

Route 53 probes the primary every 10 seconds and flips DNS on its own when it fails. A human then promotes the RDS read replica and scales the dormant Auto Scaling group — both through one script. The same application serves from the second region, under the same URL.

The client never changes anything.

Why pilot light, not the other three

I picked the pattern from the recovery objectives, not the other way around.

My targets were explicit:

• RPO under 1 minute (in practice: seconds of async lag)
• RTO under 30 minutes (expected: 10 to 15, dominated by RDS promotion)

Backup and restore misses the RTO. Active-active is a different product: Aurora Global Database or application-level conflict handling. Warm standby looks tempting until you notice the long pole is not instance boot — it is replica promotion. Paying for idle compute does not shorten that.

Pilot light is the cheapest pattern whose RTO is measured in minutes, not hours.

What actually sits in each region

Both regions are built from the same Terraform modules. They are structurally identical:

• VPC across two AZs. Public subnets hold the ALB and one NAT. Private subnets hold the API and RDS. Nothing else gets a public IP.
• ALB terminates TLS with a regional ACM certificate, redirects HTTP → HTTPS.
• Auto Scaling group of Amazon Linux 2023. No SSH. Session Manager only.
• RDS PostgreSQL + S3 for attachments + Secrets Manager for the DB password.

The only differences:

1. Which database is writable
2. The ASG desired capacity: 2 in Ireland, 0 in Paris

That single number is the pilot light.

Two replication streams, one irreversible step

S3 and Secrets Manager can be live in two places. A writable relational database cannot — not without Aurora Global Database, which I deliberately did not use.

So the runbook exists mostly for one action: promotion. Everything else is already there.

The failover chain

This is the part diagrams skip.

1. /health runs SELECT 1 against the local database. Broken data layer → failing readiness.
2. The ALB target group uses /health. Instances that cannot reach the DB leave rotation.
3. Route 53 probes /health through the primary ALB every 10 seconds. Two consecutive failures mark the record unhealthy.
4. DNS flips to the secondary ALB. About 30 seconds plus TTL. No human.
5. A CloudWatch alarm pages someone through SNS.
6. That someone runs ./scripts/failover.sh: promote the replica and scale the ASG in parallel.
7. New instances come up against a now-writable database, pass /health, and Paris serves.

The secondary DNS record does not evaluate target health. While the pilot light is still warming, Route 53 would otherwise have zero healthy answers. Sending clients to a region that is about to be ready beats sending them nowhere.

Verify with the header the API already returns:

curl -si https://<your-domain>/health | grep -i x-serving-region