Chaos Engineering in Practice
Deliberately injecting failure into production to verify that resilience mechanisms work — an experiment with a hypothesis, not vandalism.
Definition
Chaos engineering runs controlled experiments against a system to test whether it behaves as designed under failure. The essential elements are a hypothesis, a blast radius limit, a stop condition, and a measurement.
"We turn things off randomly" is not chaos engineering. "We believe that terminating one instance of service X will not affect p99 latency, and we will verify that on 1% of traffic with an automatic abort if error rate exceeds 0.5%" is.
Why it is necessary
Resilience mechanisms that are never exercised do not work. Timeouts are misconfigured, fallbacks throw exceptions, circuit breakers are set to thresholds that never trip, retry logic amplifies rather than recovers, and the runbook references a dashboard that was deleted.
None of this is discoverable by reading code or by testing in an environment that does not resemble production. The only way to know a degradation path works is to make it run.
Industry example
Netflix's practice — beginning with randomly terminating instances and extending to injecting latency, failing dependencies and exercising whole-region evacuation — is the origin of the discipline, and the part most often missed is the direction of causation.
Chaos engineering did not make Netflix resilient. It verified and maintained resilience that had been designed in: bulkheads, standardised timeouts, per-dependency fallbacks, and client-side resilience libraries. Injecting failure into a system without those mechanisms produces outages, not learning.
The correct sequence is: design the resilience, then verify it continuously, then automate the verification so regressions are caught. Organisations that start by injecting failure into a fragile system correctly conclude that chaos engineering is dangerous.
Progression
- In non-production, to build tooling and confidence.
- In production, small blast radius, business hours, humans watching.
- Automated and continuous, with automatic abort.
- Large-scale exercises — zone evacuation, region failover — as scheduled events.
What to inject, roughly in order of value
Latency (far more common in reality than hard failure, and far more damaging), dependency errors, instance termination, resource exhaustion, network partition, zone loss.
Latency injection is the highest-value and least-used. Most cascading failures begin with a slowdown, not an outage, and a system tested only against hard failures has not been tested against its most likely incident.
Prerequisites
Good observability, or you cannot tell what happened. A stop button that works. Stakeholder agreement. And enough baseline resilience that the experiment tests something rather than proving fragility.
Interview question
"What is the first chaos experiment you would run on a system you have just inherited, and what would you need in place before running it?"