Leadership wants to start chaos engineering. What is your first experiment, and what must be true before you run anything?
Show the full answer Hide the answer
What the interviewer is testing
Whether you know the preconditions, and whether you design an experiment as a hypothesis rather than a disruption.
What must be true first
Observability good enough to see the effect. An experiment you cannot measure is an outage you caused. You need per-dependency latency and error rates, and the ability to see user impact within minutes.
No known unaddressed reliability problems. If the team already knows the system falls over when the cache is cold, running an experiment to confirm it buys nothing. Fix the known list first.
An abort mechanism that is tested, and a defined abort criterion agreed in advance.
Awareness — announced window, the on-call engineer informed, and a clear way to distinguish experiment effects from a real incident occurring simultaneously.
The first experiment
Start with a non-critical dependency in a non-production environment, with a written hypothesis:
"If the recommendations service becomes unavailable, the product page still renders within 500 ms with a static fallback, checkout error rate does not change, and an alert fires within two minutes."
Then inject latency — not failure. Slow is the more common and more damaging failure mode, and systems handle unavailability far better than degradation, so latency injection finds more.
What you will probably find
Consistently, across organisations: the fallback exists in code but was never wired up; the timeout is 30 seconds rather than 300 ms; a dependency believed optional is on the critical path; or the failure is handled correctly and nobody is alerted, which is its own finding.
What a strong answer adds
The progression: non-production, then production with a small blast radius during business hours with the team watching, then scheduled game days, then automated continuous experiments. Skipping to production automation is how chaos engineering gets banned after one incident.
Common weak answers
Terminating random production instances as a first experiment. Running without a written hypothesis, which produces an anecdote rather than a finding.