A regional mobility platform in Grab's mould runs weekly chaos experiments in staging. They find nothing new any more. The team wants to run in production against live dispatch. Sequence the move so each step is reversible, and name the point of no return.

Why staging stopped finding things

Because it has no neighbours, no real data distribution and no real concurrency, so the only failures it can surface are the ones already understood. An experiment programme that finds nothing for a quarter is not evidence of resilience; it is evidence the environment has been exhausted.

The sequence

1. Establish the prerequisites, before any production experiment. Named: a user-journey SLO with a measured error budget, an abort mechanism that does not depend on the component being tested, an incident process that treats an experiment-caused incident like any other, and a pre-registered hypothesis per experiment. Without the pre-registered hypothesis, an experiment that finds nothing is indistinguishable from an experiment that was not looking.

2. Observe in production without injecting. Run the experiment's measurement path against normal traffic for a week and confirm the signals move when something real happens. This is the step that catches a broken measurement before it is relied on. Fully reversible; nothing is injected.

3. Read-only, one cell, off-peak. Inject latency on a read path, in a single cell, outside peak hours. A cell boundary gives a bounded, nameable population who can be affected, decided in advance — which is what makes the experiment approvable. A 1% traffic slice does not: it is 1% of every customer's requests, so everybody sees some failures.

4. Add the automatic abort, tied to burn rate. The stop condition is error-budget burn rate over a short window, not a fixed error-count threshold. A fixed threshold either trips on a small cell's normal noise or fails to trip on a real regression; burn rate normalises for traffic and is comparable across cells. Verify the abort by deliberately triggering it.

5. Read-only, one cell, during peak. The same experiment where the system is actually loaded, which is where the interesting couplings live.

6. Expand the fault types on read paths: latency, then error injection, then dependency unavailability. One new fault class at a time, each starting off-peak in one cell.

7. The first write-path experiment. This is where the sequence slows down and needs a named authoriser, a rehearsed data-repair procedure, and a reconciliation check that runs afterwards.

8. Expand across cells, then make it continuous.

The point of no return

Step 7: the first experiment that affects a write path. Everything before it can be aborted with the system returning to its prior state automatically. A write-path experiment can leave inconsistent data — a dispatch assigned twice, a payment recorded without its counterpart — and rolling back the experiment does not roll back the data.

That makes the gate for step 7 specific: a tested repair procedure for the inconsistency class the experiment can produce, not a general confidence that things will be fine.

Where it can go wrong, and how you would know

Cell affinity leaking. A shared cache, a global rate limiter, a cross-cell lookup or a shared database quietly reconnects the cells, so an experiment in one affects all. The diagram says the cells are independent; only a game day says whether they are. Verify by injecting a hard failure in one cell off-peak and confirming the others' SLIs do not move.

An abort that depends on the failing component. If the abort switch is served by the service being degraded, there is no abort. This is the same circular-dependency failure that makes real incidents long.

How long it really takes

Steps 1 and 2 are a few weeks, mostly spent discovering that the error budget is not actually measured per journey. Steps 3 to 6 are a quarter at one new fault class every two weeks. Step 7 is scheduled separately and often lands a quarter later. A team that reaches continuous production experiments within a year has moved quickly.

When not to start at all

If the last three incidents were caused by known, unfixed reliability issues. Chaos engineering finds unknown weaknesses, and spending it to rediscover weaknesses already on the backlog is the most expensive possible way to write a ticket. Fix the known ones first; the programme will find more afterwards.