What is the first chaos experiment you would run on a system you have just inherited, and what must be in place first?

What is being tested

Whether you know the prerequisites, and whether you pick the highest-value experiment rather than the most dramatic one.

What must be in place first

Observability sufficient to tell what happened. Without it, an experiment produces an incident and no learning. You need to see error rates, latency percentiles, and the behaviour of the specific dependency you are perturbing.

A stop button that works, and has been tested. Not a plan to redeploy — an immediate abort.

Stakeholder agreement, including support, so the exercise is not mistaken for a real incident.

A hypothesis. "We believe terminating one instance of service X will not affect p99 latency." An experiment without a prediction is not an experiment.

Enough baseline resilience that the experiment tests something. This is the prerequisite most often skipped, and it is why the ordering of the practice matters.

The critical sequencing point

Chaos engineering did not make Netflix resilient — it verified and maintained resilience that had already been designed in: bulkheads, standardised timeouts, per-dependency fallbacks, client-side resilience libraries. Injecting failure into a system without those mechanisms produces outages, not learning, and correctly teaches the organisation that chaos engineering is dangerous.

For an inherited system: read the code first. Are there timeouts? Are there fallbacks? If the answer is no, the first work is adding them, not injecting failure to prove they are missing — you already know.

The first experiment

Latency injection on the most critical dependency, in a non-production environment first, then in production at a small blast radius.

Why latency and not termination:

• Most cascading failures begin with a slowdown, not an outage. A dependency going from 50 ms to 3 s exhausts thread and connection pools by Little's Law, and the caller then fails requests that never touched it.
• A slow dependency is far more common in reality than a cleanly dead one.
• It is the case least likely to be handled. Code that handles connection-refused often does not handle a response that takes 30 seconds.
• It reveals timeout configuration, which is almost always wrong somewhere: timeouts longer than the caller's own timeout so they never fire, retries at three layers multiplying into 27 requests, circuit breaker thresholds unreachable before the pool is exhausted.

The progression afterwards

1. Non-production, building tooling and confidence.
2. Production, small blast radius, business hours, humans watching.
3. Automated and continuous, with automatic abort, so regressions are caught.
4. Large-scale scheduled exercises — zone evacuation, region failover.

What a strong answer adds

Naming what an experiment failing means. A failed experiment is a success — it found a real weakness cheaply, at a time of your choosing, with everyone watching, rather than at 3am on a holiday. Framing it that way with stakeholders in advance is what determines whether the practice survives its first genuine finding.