A mobility platform wants to start chaos engineering. Its dispatch system is business-critical and the team is nervous. What is a responsible starting sequence, and what must exist first?
Show the full answer Hide the answer
What must exist first
Chaos engineering without these produces incidents rather than learning:
1. Observability good enough to detect the impact. If you cannot tell within a minute that the experiment is causing harm, you cannot run it. This prerequisite disqualifies more organisations than any other, and discovering it is itself valuable.
2. A tested abort mechanism. A single action that stops the experiment immediately, verified before the first run.
3. A hypothesis. "We believe the dispatch service will continue matching riders when the ETA service is unavailable, with degraded accuracy." An experiment without a prediction is not an experiment; it is an outage with a positive attitude.
4. Agreement on blast radius, with the business, in advance.
The sequence
Stage 1 — non-production, with realistic load. Confirm the tooling works and the abort mechanism functions. Learn nothing about production; that is fine, this stage is about the process.
Stage 2 — production, single non-critical dependency, tiny blast radius. Degrade one optional dependency for a small percentage of traffic in one region during a low-traffic window. For a mobility platform, something like the surge-pricing display or a recommendation surface — visible if it breaks, not harmful.
Stage 3 — production, critical dependency, still bounded. Now the interesting experiments. Inject latency into the ETA service for 1% of dispatch requests. Does dispatch degrade gracefully to a cruder estimate, or does it block? This is where the real findings are, and the blast radius stays small.
Stage 4 — infrastructure-level. Remove an instance, degrade a zone, partition a network path.
Stage 5 — continuous, automated, during business hours. The mature state. Experiments run regularly so regressions are caught, and running during working hours is deliberate: it means the people who can respond are present.
The findings that matter most
In practice the highest-value discoveries are rarely "the system failed". They are:
- Timeouts that are too long, so a degraded dependency consumes the caller's entire budget.
- Fallbacks that do not work because they were never exercised, or because they share a dependency with the primary.
- Hidden hard dependencies — a service believed optional that is actually on the critical path.
- Retries that amplify rather than recover.
- Alerting that does not fire, which is arguably the most valuable finding of all, since it means real incidents are also going undetected.
The thing to say to a nervous team
These failures will happen anyway. The choice is not whether the ETA service will be slow one day; it is whether you discover the consequences at 2 p.m. with the team watching and a kill switch ready, or at 3 a.m. during peak demand with no warning.