Netflix: Chaos Monkey and the Simian Army
also called Simian Army, Chaos Kong
Netflix deliberately terminated production instances during business hours to force engineers to build for failure rather than hope against it.
The problem
In 2008 a database corruption halted Netflix's DVD shipping for three days. The lesson they drew was not "buy better hardware" but that vertically-scaled systems with single points of failure were the wrong shape. The migration to AWS that followed took until January 2016 — seven years — and it traded one problem for another: cloud instances fail routinely, unpredictably, and without warning.
The organisational difficulty was that engineers knew this and still wrote code assuming instances stayed up, because the failures were rare enough to be someone else's problem.
What they did
Chaos Monkey randomly terminates production instances during working hours, when engineers are present to observe and fix. That timing is the design decision people miss: the point is not to test resilience, it is to make instance failure so routine that nobody can ship code that assumes otherwise.
It expanded into the Simian Army — Latency Monkey injecting delays, Conformity Monkey finding instances that violated standards, Chaos Gorilla taking out an availability zone, Chaos Kong evacuating an entire AWS region. Later this matured into automated experiments running continuously with control and canary populations rather than random destruction.
The trade-off
This only works because Netflix's domain tolerates it. A failed request for a recommendation row degrades the page; it does not lose money irrecoverably. Netflix pairs it with aggressive fallbacks — a personalised row that fails is replaced by a static one, and the customer sees a slightly worse homepage rather than an error.
It also demands real investment before it is safe: strong observability, fast rollback, and no known unaddressed reliability problems. Running chaos experiments against a system you already know is fragile buys nothing.
The transferable lesson
The mechanism that matters is making failure routine rather than exceptional, and that is available at any scale. A quarterly game day, a scheduled dependency-latency injection in staging, or simply terminating one instance during business hours will surface the same class of finding: the fallback that exists in code but was never wired up, the 30-second timeout, the "optional" dependency that is on the critical path.
What does not transfer is starting in production. Netflix earned that position over a decade.