GCP The Hard Way — Part 8: Designing a Broken System for Someone Else to Debug

Introduction

Every prior post in this series involved diagnosing a failure someone
else (in this case, the author) had designed. This final post reverses
that relationship: you design the failure. Constructing a realistic,
diagnosable incident requires a deeper understanding of a system than
fixing one does, because you must accurately predict which symptoms a
given misconfiguration will produce. This exercise is a common
technique in SRE training programs, and this post walks through
applying it to a Google Cloud environment.

Solution overview

┌─────────────────────┐        ┌────────────────────┐
│   Design Phase        │        │    Debug Phase       │
│                        │───────▶│                       │
│  - Inject 3–5 faults   │ hand   │  - Investigate via    │
│  - Write incident      │  off   │    logs and metrics   │
│    ticket (symptoms     │        │  - Isolate root cause │
│    only, no root cause) │        │  - Remediate           │
└─────────────────────┘        └────────────────────┘

Criteria for a well-designed failure scenario

1. Clear symptom, unclear cause — the observer sees a concrete behavior, not an obvious misconfiguration
2. Reliably reproducible — the failure occurs consistently, not intermittently by chance
3. Plausible red herrings present — realistic incidents involve ruling out multiple hypotheses, not confirming the first guess
4. A definitive verification method exists — once the correct hypothesis is identified, there must be an unambiguous way to confirm it

Walkthrough

Step 1: Establish a known-good baseline

Reuse the three-tier architecture from Part 2 (Load Balancer → MIG →
Cloud SQL), and confirm it functions correctly before introducing any
faults — this establishes a clean baseline to distinguish injected
faults from pre-existing configuration issues.

Step 2: Select and inject 3–5 faults

A representative fault bank, spanning difficulty levels:

Example injection:

gcloud sql instances patch app-db --database-flags=max_connections=20

Choose faults spanning multiple categories (network, IAM, database,
application) rather than concentrating on one area, to keep the
diagnostic exercise varied.

Step 3: Write the incident ticket

The ticket should be written from the perspective of an end user
reporting a problem, deliberately omitting any technical language that
would hint at the root cause:

# Ticket #001: Intermittent login failures during peak hours

## Description
Users report occasional "connection timeout" errors when logging in,
primarily between 9–10 AM. No issues reported in the afternoon.

## Already investigated
- VM restarted — issue persists
- CPU and memory utilization checked — both within normal range

## Access provided
Cloud Console (read-only), Cloud Logging, gcloud CLI

Including one or two already-ruled-out hypotheses ("already
investigated") makes the exercise more representative of real
incident response, where an initial theory is often incorrect.

Step 4: Grant scoped access for the investigator

gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member="user:<investigator-email>" \
  --role="roles/viewer"

gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member="user:<investigator-email>" \
  --role="roles/logging.viewer"

Read-only access prevents accidental remediation before the
investigator has confirmed a root cause.

Step 5: Structured investigation approach

Whether debugging as a peer or returning to your own system after a
delay, apply a consistent process:

1. Reproduce the symptom first — do not make changes before confirming the reported behavior
2. Narrow scope by elimination — isolate whether the fault sits in the network, application, or database layer by testing each layer independently (e.g., curl localhost on the VM before testing through the load balancer)
3. Read logs systematically — filter Cloud Logging by severity and correlate timestamps with reported symptom occurrence
4. Verify each hypothesis definitively before moving to the next, rather than changing multiple variables simultaneously

Comparing outcomes

| Injected fault | Identified? | Time to identify | False leads pursued |
|---|---|---|---|
| Fault 1 | | | |
| Fault 2 | | | |

Clean up resources

gcloud projects delete <PROJECT_ID>

Conclusion

This series has moved through the core operational competencies of
infrastructure engineering: provisioning without code changes,
diagnosing availability failures, auditing access control, choosing
data stores appropriate to workload characteristics, shipping releases
safely, validating disaster recovery assumptions, controlling cost, and
finally, designing failure scenarios well enough to teach them to
others. That last skill — being able to construct a realistic incident,
not just resolve one — is a strong signal of systems-level understanding,
and a capability directly transferable to building internal SRE
training programs.

This concludes the Google Cloud Fundamentals: The Hard Way series.