Surviving the Chaos: Injecting Faults with the ChaosEngineeringMasterDeck

Abstract: How we turned "will it hold up?" into testable hypotheses using a deck of chaos experiments (the ChaosEngineeringMasterDeck), and what we uncovered when degrading the latency of a payment microservice in a real-world environment.

1. Why Break Things on Purpose In distributed systems, partial failure is the normal state, not an exception. With dozens of services, each with 99.9% availability, the composite availability of a request traversing 10 of them in series is approximately $0.999^{10} \approx 99.0\%$. That represents roughly 87 hours of downtime per year that no one anticipated by evaluating services individually. Unit and integration tests verify what code does when everything works; they reveal nothing about system behavior when the network slows down, a node disappears, or a downstream dependency stalls. Chaos Engineering is the discipline of experimenting on a system to build confidence in its capability to withstand turbulent conditions. Formalized in the Principles of Chaos Engineering (born out of Netflix's experience with Chaos Monkey), it follows the empirical scientific cycle:
2. Define steady state via observable business or system metrics (not just "the server is up").
3. Formulate a hypothesis: the steady state will persist despite fault $X$.
4. Inject realistic faults reflecting real-world events (latency, packet loss, node or availability zone outages).
5. Attempt to refute the hypothesis by contrasting control and experimental groups.
6. Contain the blast radius and abort immediately if safety thresholds are breached. The Mindset Shift The traditional approach focuses on preventing failures (maximizing MTBF). The resilience approach assumes failures are inevitable and optimizes for recovery time and graceful degradation (minimizing MTTR and user impact). In practice, this reframes the question from "How do we prevent the payment service from going down?" to "What does the user experience when it goes down, and how quickly do we recover?" Regarding staging versus production: mature implementations target production, as only production reflects real traffic, authentic data, and live configurations. However, the correct progression is staging first, followed by production with a minimal blast radius. An experiment should only transition environments once automated rollback and abort triggers are thoroughly validated.
7. Dissecting the ChaosEngineeringMasterDeck In our context, the Master Deck is a version-controlled catalog of chaos experiments modeled as playing cards, maintained in Git alongside application code. The card deck analogy serves a practical operational purpose: each card represents a self-contained unit that can be drawn during a Game Day, requiring mandatory fields. If any required field is empty, the card cannot be played.

Why This Format?
The value lies not in the novelty of the "deck" metaphor, but in enforcing explicit hypotheses and strict abort criteria. The most common pitfall in chaos experiments is executing without a hypothesis: injecting a fault, observing arbitrary behavior, and drawing unfounded conclusions. The deck enforces scientific rigor upfront. Stored in Git, each card undergoes code review just like production changes.
Example Card in YAML

id: REDE-03
titulo: Latência alta no serviço de pagamento
naipe: Rede
hipotese: >
  Com 2000 ms de latência adicional em payment-service para 5% do tráfego,
  o checkout mantém p99 < 800 ms e taxa de erro < 1%.
variaveis:
  independentes: { delay_ms: 2000, trafego_afetado: "5%", duracao: "10m" }
  dependentes: [p99_checkout, erro_5xx, threads_ativas, estado_circuit_breaker]
metricas_controle:
  p99_checkout_ms: { max: 800 }
  erro_5xx_pct: { max: 1.0 }
raio_explosao: { escopo: "namespace staging-canary", trafego: "5%", duracao_max: "10m" }
aborto:
  - "erro_5xx_pct > 3 por 60s"
  - "conversao_checkout cai > 2 p.p."
padroes_esperados: [circuit-breaker, timeout, fallback]

1. Case Study: Card NET-03 Architecture Under Test An e-commerce platform running on AWS with the following checkout flow: • API Gateway $\rightarrow$ order-service (Spring Boot 3, Java 21) $\rightarrow$ payment-service (Spring Boot) $\rightarrow$ External Payment Provider. • order-service persists state to Amazon RDS (PostgreSQL, Multi-AZ). • Workloads run on Amazon EKS with Horizontal Pod Autoscaler (HPA), distributed across three Availability Zones. • Observability powered by Micrometer + Prometheus + Grafana, with distributed tracing via OpenTelemetry. The critical failure domain is the synchronous invocation order-service $\rightarrow$ payment-service. It is a textbook candidate for cascading failure: if payment stalls, worker threads in order-service become blocked waiting for responses, starving the connection and thread pools until an otherwise healthy upstream service stops serving traffic entirely. Hypothesis With an additional 2000 ms of latency injected into 5% of traffic destined for payment-service, checkout maintains p99 < 800 ms and 5xx errors < 1%, because the circuit breaker will trip open and the fallback mechanism will enqueue the transaction for asynchronous processing. Fault Injection Using Chaos Mesh (equivalent to AWS Fault Injection Service, Litmus, or Toxiproxy):

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: rede-03-latencia-pagamento
  namespace: staging-canary
spec:
  action: delay
  mode: fixed-percent
  value: "5"
  selector:
    namespaces: [staging-canary]
    labelSelectors:
      app: payment-service
  delay:
    latency: "2000ms"
    jitter: "200ms"
  direction: to
  duration: "10m"

Blast Radius & Containment
The blast radius is strictly bound along three dimensions prior to execution:
• Scope: Restricted exclusively to the staging-canary namespace; zero impact on other application pods or databases.
• Traffic: 5% of requests, isolated via header-based routing at the API Gateway.
• Duration: 10 minutes, configured natively in the resource definition so that if connection to the cluster is lost, the experiment terminates automatically.
The emergency abort command (kubectl delete networkchaos net-03-payment-latency) was tested beforehand, complemented by automated abort webhooks triggered by Alertmanager whenever safety thresholds were violated.

1. Observability and Results Telemetry Tracked

Note: Business metrics validate technical indicators. A degraded system might maintain low latency and low error rates by prematurely returning an empty screen, all while failing 100% of revenue-generating transactions.
Round 1: Hypothesis Refuted
During the first run, the hypothesis failed. Illustrative experimental data:

Distributed tracing quickly isolated root causes:

1. The underlying HTTP client (RestTemplate) lacked an explicit read timeout (defaulting to infinite).
2. The Resilience4j circuit breaker was only configured to trip on explicit application exceptions, not slow calls. Consequently, 2-second calls were never registered as failures. The circuit breaker remained closed while Tomcat's worker threads rapidly exhausted the pool. This yielded the primary architectural insight: a resilience pattern was declared in code, yet remained completely inoperative under real failure dynamics. Remediation We deployed three targeted fixes:

# application.yml (Resilience4j)
resilience4j:
  circuitbreaker:
    instances:
      payment:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 20
        minimumNumberOfCalls: 10
        failureRateThreshold: 50
        slowCallDurationThreshold: 800ms   # chamada lenta passa a contar
        slowCallRateThreshold: 50
        waitDurationInOpenState: 15s
        permittedNumberOfCallsInHalfOpenState: 3
  timelimiter:
    instances:
      payment:
        timeoutDuration: 1s
  bulkhead:
    instances:
      payment:
        maxConcurrentCalls: 20             # isola o pool de threads

@CircuitBreaker(name = "payment", fallbackMethod = "enfileirarPagamento")
@Bulkhead(name = "payment")
public PagamentoResultado cobrar(Pedido pedido) {
    return paymentClient.cobrar(pedido);
}

// Fallback: não perde o pedido; marca como PAGAMENTO_PENDENTE e publica em fila
private PagamentoResultado enfileirarPagamento(Pedido pedido, Throwable t) {
    filaPagamentos.publicar(pedido.getId());
    return PagamentoResultado.pendente(pedido.getId());
}

Additionally, low-level socket and connect timeouts (500 ms connect, 1000 ms read) were set on the HTTP client pool, as TimeLimiter does not guarantee cancellation of blocking I/O calls on underlying threads.
Round 2: Hypothesis Upheld

The architecture maintained resilience: the circuit opened quickly, the bulkhead prevented thread pool exhaustion, and the asynchronous fallback preserved incoming orders. The engineering tradeoff was deliberate and accepted by business stakeholders: a small fraction of customers received an immediate "payment processing" notification rather than a hard checkout failure.
Resilience Patterns in Action
• Timeout: Enforces a ceiling on wait times, shedding resources promptly.
• Circuit Breaker: Fails fast against degraded dependencies, preventing backlog build-up.
• Bulkhead: Isolates critical resources (thread pools, concurrency quotas) per dependency.
• Queue-based Fallback (Graceful Degradation): Trades immediate strong consistency for system availability.
• Auto Scaling (HPA): Proved ineffective during the incident. Because the bottleneck was blocked I/O rather than CPU saturation, horizontal scaling would have merely amplified concurrency pressure against the already degraded payment backend.
Other Cards in the Deck
The framework accommodates wider failure domains, each governed by its own blast radius constraints:
• AZ-01 (Complete Availability Zone Outage): Verifies traffic reroutes cleanly across surviving AZs and confirms AWS Aurora / RDS Multi-AZ failover completes within SLA (typically 60–120s; actual failover time must be empirically measured rather than assumed). Executed via AWS FIS.
• DATA-02 (Database Connectivity Loss): Validates exponential backoff with full jitter on retries, alongside HikariCP connection pool recovery mechanics when pool connections become stale.

1. Conclusion Key Takeaways
2. Unverified resilience is not resilience. The circuit breaker existed in source control, yet failed to protect the system because it had never been tested against latency-based degradation.
3. Slowness is worse than outright failure. Downstream crashes fail fast; downstream latency silently exhausts upstream threads and memory.
4. Hypotheses must be falsifiable and validated by business metrics. Relying solely on technical signals masks functional edge cases.
5. Blast radiuses and automated aborts are prerequisites, not polish. They establish the safety boundary required to run experiments routinely.
6. Every experiment must drive traceable action: a bug fix, an alert adjustment, or an updated runbook, followed by re-executing the card to validate remediation (as shown in Round 2). Cultural Impact on DevOps The most enduring payoff of chaos engineering is cultural. By treating failure as an objective target of empirical study rather than human error, teams shift toward proactive architectural reviews and blameless post-mortems. Engineers begin developing with failure modes in mind ("What happens when this API call hangs?"), while operations teams validate runbooks in realistic conditions. Game Days double as practical incident-response training, cutting Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR). Next Steps: Automate low-risk cards inside CI/CD pipelines for continuous staging execution, reserve high-impact cards for scheduled Game Days, and track maturity through concrete KPIs: MTTR reduction, issues identified per experiment, and first-round card pass rates.