Week 22: Chaos Engineering & Resilience Testing

1. The Hypothesis-Experiment Loop

Chaos engineering isn't "randomly break things and see what happens" — done well, it's a disciplined loop borrowed from the scientific method. Start from a specific, falsifiable hypothesis about steady-state behavior: "if the inventory-service primary Pod is killed, checkout-service's p99 latency stays under 500ms because of the retry and circuit-breaking config from Week 16." Then design the smallest experiment that could disprove it, run it against a real (ideally production-like, and eventually production) environment, and compare what actually happened against the hypothesis.

The order matters: define the hypothesis and the steady-state metric before running the experiment, not after — deciding what "success" looks like retroactively, once you already know the outcome, isn't a real test of anything. Every experiment should also have a defined blast radius and an automatic abort condition, so a genuinely bad outcome doesn't turn a controlled experiment into a real incident.

2. Fault Injection with Chaos Mesh

Chaos Mesh is a Kubernetes-native chaos engineering platform, installed via Helm, that expresses fault injection as ordinary Kubernetes custom resources — the same CRD pattern from Week 15's Operators, applied to deliberately breaking things instead of running them.

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: inventory-pod-kill
spec:
  action: pod-kill
  mode: one                      # exactly one matching Pod per trigger
  selector:
    namespaces: [production]
    labelSelectors:
      app: inventory-service
  scheduler:
    cron: "@every 10m"

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: inventory-network-delay
spec:
  action: delay
  mode: all
  selector:
    namespaces: [production]
    labelSelectors: { app: inventory-service }
  delay:
    latency: "300ms"
    jitter: "50ms"
  duration: "5m"
  direction: to
  target:
    selector:
      labelSelectors: { app: checkout-service }
    mode: all

PodChaos tests the failure Weeks 11–12 designed for directly — does a killed Pod actually get replaced and does traffic actually route around it during the gap. NetworkChaos tests something readiness probes can't: not "is the dependency down," but "is the dependency slow," which is what Week 16's timeout and perTryTimeout settings were specifically written to survive — this experiment is the direct verification that they actually do.

3. Game Days & Turning Findings Into Fixes

A game day is a scheduled, team-wide chaos exercise — not a solo engineer running an experiment quietly, but the whole on-call rotation from Week 21 actively responding to an injected failure as if it were real, using their actual runbooks and dashboards. The goal is as much about testing the humans and the process as it is about testing the system: does the runbook's first step actually work, does the right person get paged, does the burn-rate alert from Week 21 fire when it should.

A game day always ends the same way Week 21's incidents do — with a written report. The difference is a chaos game day's report has one extra, valuable property: because the failure was deliberately injected, the "ground truth" of what happened is known exactly, which makes it unusually easy to spot every place the runbook, the alert, or the system's actual behavior diverged from what the team expected.

Hypothesis: checkout-service p99 stays under 500ms during an
inventory-service Pod kill.

Result: PARTIALLY CONFIRMED
- p99 peaked at 640ms for ~8 seconds during Pod rescheduling. FAIL.
- Circuit breaker (Week 16) correctly ejected the failing endpoint after
  5 consecutive errors, as designed. PASS.
- Burn-rate alert (Week 21) did NOT fire -- the 8-second spike was too
  short to trip even the 5m window. Alert may be missing short,
  high-severity spikes entirely.

Follow-ups:
1. Add a readiness probe `initialDelaySeconds` tuning pass -- new Pod
   was marked Ready before its connection pool warmed up. (owner: X)
2. Add a shorter secondary burn-rate window (1m) for spike detection. (owner: Y)
3. Re-run this exact experiment after both fixes ship, to confirm.

Notice the report doesn't stop at "it mostly worked" — it surfaces a real gap (the alerting window entirely missed a genuine spike) that no amount of reading the Prometheus rule file in Week 21 would have caught, because that gap only becomes visible when a real, timed failure actually happens against it.

4. Hands-on Exercise

5. Knowledge Check

Three quick questions. Expand each to check your answer.