A change must reach 30 regions. Policy requires a 20-minute bake with no regression before the next step. Rolling out one region at a time therefore takes 10 hours. Leadership wants it under 3 hours without abandoning the bake. Which schedule meets that, and what does it cost?
Show the full answer Hide the answer
The assumptions, stated
- 30 regions, 20-minute bake between steps, bake is wall-clock and runs concurrently for all regions in a step.
- Deployment time per region is small relative to the bake; ignore it.
- The bake's job is to expose a regression in the regions already carrying the change before more regions are added.
The arithmetic
Sequential: 30 steps × 20 minutes = 600 minutes = 10 hours. This is the stated baseline.
Exponential rings: 1, 2, 4, 8, 15 regions. That is 5 steps, so 5 bakes × 20 minutes = 100 minutes, comfortably inside 3 hours. Cumulative exposure after each bake: 1, 3, 7, 15, 30 regions.
The property that makes this work is that the number of steps grows with the logarithm of the region count while exposure grows geometrically. Doubling 30 regions to 60 adds one step — 20 minutes — rather than another 10 hours.
The cost is concentrated in the final step. The last ring exposes half the fleet at once, and it is the only step with no bake after it. Everything riding on this schedule depends on a regression being detectable within 20 minutes in a small number of regions, which is the assumption to test rather than assume.
Which assumption dominates the error
Whether 20 minutes is long enough to surface the regression class you care about. A latency or error-rate regression appears in minutes; a memory leak, a disk-fill, a certificate issue or a data-corruption defect does not. For those, no ring schedule helps and the honest answer is that the bake is not the right control — a longer soak in one region is, and it cannot be compressed.
The second assumption is regional homogeneity. Rings are safe only if an early ring is representative. If region 1 is small and region 29 is the only one with a particular dependency or regulatory configuration, the schedule tested nothing relevant until the final step.
Why the other options fail
Halve the bake to 10 minutes. It halves the detection window to meet a schedule, which is trading the control for the thing the control exists to prevent. It also gets to 5 hours, still missing the target. When the schedule and the safety mechanism conflict, changing the mechanism's threshold is the answer to avoid.
All 30 in parallel after one canary region. Two steps and 40 minutes, and the blast radius goes from 1 region to 30 in one move. A defect that is only visible at scale, or in a region unlike the canary, reaches everyone simultaneously. This is the schedule that produces the incidents rings exist to prevent.
Three parallel groups of 10. This reads as progressive and is not: the first step exposes 10 regions, a third of the fleet, with nothing smaller before it. Parallel groups reduce total time by reducing steps, and they do it by making the first exposure large, which is the opposite of what a staged rollout is for.
When not to use rings at all
When the regression class you fear does not surface in 20 minutes. A memory leak, a disk-fill, a certificate expiry or a slow data-corruption defect needs a soak in one region measured in days, and no ring schedule compresses it. Prefer a long soak in a single representative region for those changes, and rings only for changes whose failure is visible in error rate, latency or saturation.
What the schedule still needs
An automatic abort tied to a signal, and ordering by risk. Rings should be ordered so early ones are representative but low-traffic, and the abort should trigger on error-budget burn rate rather than on a fixed error threshold, so a small region's noise does not stop the rollout and a real regression does. Without an automatic abort, a 100-minute schedule simply reaches everyone faster.