Review this resilience programme. Chaos experiments run weekly in staging at 03:00, they inject only instance termination, results are recorded in a spreadsheet, and there is a kill switch that has never been used. What would you change, and what would you keep?

What is actually required

A resilience programme exists to find the difference between how a system is believed to behave under failure and how it does behave. Everything in the description optimises for not disturbing anyone, which optimises away the only thing that generates that difference.

What I would change

Move it out of 03:00. An experiment at 03:00 has no humans watching, so it tests the automation and nothing else. The most valuable outputs of a chaos programme are the ones involving people: whether the alert fires, whether it reaches someone, whether the runbook matches reality, whether the dashboard shows the problem. Run during working hours, announced, with the owning team present.

Move it towards production, carefully. Staging has different data volumes, different cache states, no real traffic mix and usually a scaled-down topology, so the failures it finds are the ones you would have predicted anyway. The path is graduated: staging, then production with a small blast radius on a non-critical path, then production with real traffic. Each step needs a way to halt.

Replace instance termination with the failures that actually happen. Instances dying is the failure modern platforms handle best, because orchestrators were built for it. The failures that cause incidents are slow dependencies, partial failures, expired credentials, full disks, DNS problems, dependency error spikes and configuration pushes. Latency injection alone will find more than a year of instance terminations.

Start every experiment with a written steady-state hypothesis. "While one instance is terminated, checkout success rate stays above 99.5% and p99 stays under 800 ms." Without a stated hypothesis and a metric, the outcome is "it seemed fine", which is why the results live in a spreadsheet that nobody reads. The hypothesis is what makes the experiment falsifiable, and a falsified hypothesis is the entire product of the programme.

Test the kill switch before you need it. An untested abort path is an unavailable abort path, and it is the only reason anyone will approve production experiments.

What I would keep

The weekly cadence. Regularity beats ambition here: a modest experiment every week finds more than a quarterly event that gets postponed.

The record. A spreadsheet is a poor tool and a written record is right. Improve it to hypothesis, result, and the action item with an owner, then track whether the action items get done - that number is the honest measure of whether the programme is worth running.

How I would argue this in the review

Ask one question: name a production incident in the last year that this programme would have caught. If the answer is none, the programme is a ritual. Then propose the smallest change with the highest yield, which is almost always latency injection into one dependency during working hours with the team watching.

When this is the wrong critique

If the platform cannot survive an instance termination yet, staging at 03:00 is the right place to find that out, and running in production would be reckless. The described programme is a reasonable starting point that has stopped developing. The failure is not where it began; it is that it never graduated.