intermediate 2 min answer

A mobility platform wants to start chaos engineering. Its dispatch system is business-critical and the team is nervous. What is a responsible starting sequence, and what must exist first?

chaos-engineeringresilience-testingblast-radiusgrabdesign
Show the full answer Hide the answer

What must exist first

Chaos engineering without these produces incidents rather than learning:

1. Observability good enough to detect the impact. If you cannot tell within a minute that the experiment is causing harm, you cannot run it. This prerequisite disqualifies more organisations than any other, and discovering it is itself valuable.

2. A tested abort mechanism. A single action that stops the experiment immediately, verified before the first run.

3. A hypothesis. "We believe the dispatch service will continue matching riders when the ETA service is unavailable, with degraded accuracy." An experiment without a prediction is not an experiment; it is an outage with a positive attitude.

4. Agreement on blast radius, with the business, in advance.

The sequence

Stage 1 — non-production, with realistic load. Confirm the tooling works and the abort mechanism functions. Learn nothing about production; that is fine, this stage is about the process.

Stage 2 — production, single non-critical dependency, tiny blast radius. Degrade one optional dependency for a small percentage of traffic in one region during a low-traffic window. For a mobility platform, something like the surge-pricing display or a recommendation surface — visible if it breaks, not harmful.

Stage 3 — production, critical dependency, still bounded. Now the interesting experiments. Inject latency into the ETA service for 1% of dispatch requests. Does dispatch degrade gracefully to a cruder estimate, or does it block? This is where the real findings are, and the blast radius stays small.

Stage 4 — infrastructure-level. Remove an instance, degrade a zone, partition a network path.

Stage 5 — continuous, automated, during business hours. The mature state. Experiments run regularly so regressions are caught, and running during working hours is deliberate: it means the people who can respond are present.

The findings that matter most

In practice the highest-value discoveries are rarely "the system failed". They are:

  • Timeouts that are too long, so a degraded dependency consumes the caller's entire budget.
  • Fallbacks that do not work because they were never exercised, or because they share a dependency with the primary.
  • Hidden hard dependencies — a service believed optional that is actually on the critical path.
  • Retries that amplify rather than recover.
  • Alerting that does not fire, which is arguably the most valuable finding of all, since it means real incidents are also going undetected.

The thing to say to a nervous team

These failures will happen anyway. The choice is not whether the ETA service will be slow one day; it is whether you discover the consequences at 2 p.m. with the team watching and a kill switch ready, or at 3 a.m. during peak demand with no warning.