Chaos Experiment
A controlled test that injects a specific failure to verify a hypothesis about the system's resilience, with a defined blast radius and abort condition.
The word "chaos" is misleading. A valid experiment has four parts: a hypothesis ("if the recommendation service is unavailable, the product page still renders within 500 ms with a generic list"), a defined blast radius, a stop condition, and a measured result.
Randomly breaking things in production is not chaos engineering; it is an outage.
Progression: start in a non-production environment to find the obvious gaps cheaply. Move to production with a small blast radius — one instance, a small traffic percentage, one cell. Increase scope only as confidence builds.
The failures worth injecting, roughly in order of value: dependency latency (usually more revealing than dependency failure, because timeouts are more often wrong than error handling), dependency failure, instance termination, zone loss, resource exhaustion (CPU, memory, disk, connections), and network partition.
The precondition that is regularly skipped: observability good enough to detect the impact. If you cannot tell what the experiment did, you have added risk without acquiring information. Fix the monitoring first — often the experiment's most valuable finding is that you could not see the effect.