A platform is migrating a live service and must manage risk. Which risks dominate, and which controls actually reduce them?
Show the full answer Hide the answer
The risks that dominate
1. Data loss or corruption. Irreversible, and the worst outcome. Everything else is recoverable.
2. Undetected behavioural difference. The new system works and produces subtly different results — a rounding difference, an ordering change, a missing edge case. Discovered later, by customers or by reconciliation, after the old system is gone.
3. Capacity difference under real load. The new system performs adequately in testing and differently under production traffic patterns — real skew, real concurrency, real cache behaviour.
4. Rollback becoming impossible because the new system has taken writes the old one has not.
5. Migration work starving live traffic — an unthrottled backfill turning a background task into an incident.
The controls that actually reduce them
Reduce blast radius, above everything. Per tenant, per region, per shard, per percentage of traffic. This is the single largest risk reduction available, because it converts every other risk from "affects everyone" to "affects a bounded set, and we noticed".
Shadowing with automated comparison, which addresses behavioural difference directly and is the only technique that finds undocumented edge cases before they matter.
Verification before deletion, with the source retained and restorable for a defined period. The cost of retaining is small and calculable; the cost of a verification gap discovered after deletion is unbounded.
A stated point of no return with a named authoriser, so crossing it is a decision rather than a discovery.
Rate-limited migration work with the ability to pause, monitored against production load.
Rollback as a configuration change, rehearsed, with a pre-agreed trigger and decision-maker — because rollback decisions made under pressure without a trigger are made late.
The control that is usually missing
A rehearsal of the full sequence, timed. Almost every migration plan contains a step that takes far longer than assumed. Finding that in a rehearsal rather than at 3 a.m. with traffic waiting is the entire value, and it is the step most often skipped because it feels redundant.
The live-service specific risk
Reconnection storms at cutover. For a platform holding long-lived connections, moving traffic means clients reconnect — and if that is not staged and jittered, the cutover produces exactly the mass reconnection event that the platform's worst incidents look like. Staged, server-scheduled reconnection is not optional here.