advanced 2 min answer

Leadership read about Chaos Monkey and wants chaos engineering in production next month. Your last three incidents were caused by known unfixed reliability issues. What do you say?

netflixchaosprerequisitesjudgement
Show the full answer Hide the answer

What the interviewer is testing

Whether you know the preconditions, and whether you will decline a fashionable practice that would currently do harm.

The response

Not yet, and here is what comes first.

The specific disqualifier is in the question: three incidents from known unfixed reliability issues. Chaos experiments produce findings, and there is no value in generating findings you already have and are not acting on. Running experiments against a system whose weaknesses are documented and unaddressed adds incidents without adding information.

The prerequisites

Fix the known list. This is the highest-value work available and it is being displaced by the more interesting proposal.

Observability sufficient to see the effect. An experiment you cannot measure is an outage you caused. You need per-dependency latency and error rates and the ability to see user impact within minutes.

Fallbacks that exist and are wired up. Netflix can terminate production instances because a failed personalisation row degrades the page. Without designed degradation, an experiment is just breaking things.

Fast rollback and an abort mechanism, tested.

What I would propose instead

Start where the ratio of learning to risk is best: dependency latency injection in a non-production environment, with a written hypothesis. Latency rather than failure, because slow is the more common and more damaging mode and systems handle unavailability far better than degradation.

Then game days on a schedule, then production experiments with a small blast radius during business hours, then automation. Skipping to production is how chaos engineering gets banned after one incident.

What a strong answer adds

Reframing what leadership actually wants — confidence that the system survives failure — and offering a faster route to it: a pre-mortem plus a review of the last three incidents for the same mechanism elsewhere in the estate. That delivers the finding in a fortnight without any new risk.

Common weak answers

Implementing it because it was asked for. Refusing on risk grounds without offering the path.