Your SRE team wants to run fault injection in production. Leadership is nervous. How do you make the case and what do you insist on?

The case

Failures happen in production whether or not they are injected. The choice is between discovering how the system responds at three in the morning during an unplanned event, or at eleven on a Tuesday with the team watching and an abort switch in hand.

The specific argument is that resilience mechanisms are untested code paths. Circuit breakers, retries, failover, degraded modes and timeouts are written, deployed and never exercised — and untested code that runs only during an incident is exactly the code you least want to be running for the first time.

What to insist on before the first experiment

A steady-state hypothesis. A measurable prediction in user terms — order success stays above 99.9%, checkout p99 stays under 400 ms — stated before anything is injected. Without it, this is breaking something to see what happens, which produces incidents rather than knowledge.

Observability sufficient to see the hypothesis in real time. If you cannot watch it during the experiment, you cannot tell whether it held and you cannot tell when to abort.

A defined blast radius. One service, one zone, a percentage of traffic — bounded in advance rather than "we will stop if it looks bad".

An abort condition and a tested abort mechanism, executable in seconds.

The team present, notified, and not during a change freeze or a peak period.

Start where the risk is lowest

Run the same experiments in a production-shaped environment first. They find the obvious failures cheaply, and they build the practice and the tooling.

Then the first production experiment should be one where you are confident of the answer — killing one instance of a service with many. A confirmation is a legitimate outcome and it earns the credibility for the harder ones.

The interesting result to expect

Frequently the finding arrives before injection: the team cannot agree on what the system will do. That disagreement is itself worth the exercise, and it is a good thing to be able to report to the nervous leadership as an early return.