How does chaos engineering differ from a resilience test, and how should it be integrated into delivery?
Show the full answer Hide the answer
The difference
A resilience test verifies a known behaviour; a chaos experiment discovers unknown behaviour.
A resilience test asserts "when the ETA service is unavailable, dispatch degrades to a cruder estimate" — a stated hypothesis with a pass or fail. It belongs in the pipeline and should run continuously.
A chaos experiment asks "what happens if the ETA service is slow" without knowing the answer, and its output is a finding rather than a verdict.
Both are valuable and they belong in different places. The mature pattern is that chaos experiments produce resilience tests: a discovery becomes an automated assertion so the behaviour cannot regress.
Integrating into delivery
1. Resilience tests in the pipeline, as automated assertions on the degradation behaviours the system promises. Failing them blocks the change, exactly like any other test.
2. Continuous low-level fault injection in production, at a small traffic share, so fallback paths are exercised routinely rather than held in reserve. A fallback that is never run does not work — the most reliable finding in resilience engineering.
3. Scheduled larger experiments during business hours, deliberately, so the people who can respond are present.
4. Findings converted into tests, which is what makes the practice compound rather than repeat.
The prerequisites that gate the whole practice
- Observability good enough to detect impact within a minute, which disqualifies more organisations than anything else — and discovering that is itself valuable.
- A tested abort mechanism.
- A written hypothesis; an experiment without a prediction is an outage with a positive attitude.
- Agreed blast radius.
The findings that recur
Rarely "the system failed". Usually: timeouts too long, so a degraded dependency consumes the caller's whole budget · fallbacks that do not work, because they were never exercised or share a dependency with the primary · hidden hard dependencies believed optional · retries that amplify · and alerting that does not fire, which is the most valuable finding of all since it means real incidents also go undetected.