practice

Progressive Rollout

also called Wave Deployment, Staged Rollout

Deploying across regions or cells in ordered waves with soak time between them, so a defect is contained to the earliest wave.

multi-regiondeploymentblast-radius

The single most valuable property of a multi-region estate during deployment is that the regions are not updated simultaneously. Simultaneous deployment converts a multi-region architecture into a single point of failure with a higher bill, and it is how several of the most-studied cloud outages actually happened.

A sensible wave order starts with an internal or synthetic-only cell, then the smallest production region, then the rest in increasing order of traffic, with the largest and most regulated last. Between waves there is a bake period long enough for slow-burning failures to appear — memory leaks, disk fill, scheduled jobs, cache expiry, certificate paths — which is measured in hours, not the five minutes people are tempted to use.

Two things must be true for the wave order to hold: version skew across regions must be safe, which constrains schema and event contracts exactly as rolling deployment does; and the pipeline must be able to stop and reverse mid-sequence, leaving the estate in a mixed but functioning state.

The tension to manage openly is that long bake periods extend lead time, so the wave plan is itself a risk-versus-speed decision that should be written down rather than defaulted to.