Soak Period

A progressive rollout that advances as soon as error rates look acceptable catches fast failures and misses slow ones. Memory growth, connection leaks, cache degradation, log accumulation and data-dependent problems appear over hours, not over minutes, and a rollout that completes in twenty minutes has observed none of them.

The soak period is the waiting time between stages, and it is the part of a rollout plan that is compressed first when there is pressure to ship.

Why it matters

It is the only mechanism that surfaces the accumulation class of failure before it reaches everyone. Every other control — error-rate halts, latency thresholds, business-metric checks — evaluates a snapshot, and a snapshot cannot see a gradient.

Implementation patterns

• Set the duration from the failure class you are trying to catch, not from convenience. If a connection leak takes four hours to become visible, a thirty-minute soak proves nothing about it.
• Monitor derivatives, not levels, during the soak. Memory at 60% is meaningless; memory rising 2% per hour is the finding.
• Evaluate against the stage's own baseline, not a global one, since a region or a customer segment with different traffic has a different normal.
• Automate the halt rather than relying on a human watching, who will not react in time and will rationalise a degradation that started slowly.
• Include a full business cycle for the last stage where the workload is cyclical — a week omits month-end, a peak, and the periodic job that only runs on Sundays.
• Make skipping it an explicit, recorded decision with a named approver, rather than something that happens by default under pressure.

Industry example

Globally distributed platforms such as Fly.io roll changes region by region, where the soak between regions is what converts a global change into a bounded experiment. The hazard the discipline exists for is specific: anything that can push a change to every location in seconds can push an outage to every location in seconds — and the soak is the deliberate friction that makes that capability safe.

Failure scenarios

• Soak compressed under pressure, so the accumulation class is never caught.
• Duration chosen by convenience rather than by the failure it targets.
• Levels monitored rather than derivatives, so a gradient is invisible.
• A global baseline applied to a region with different traffic, producing false alarms then ignored alarms.
• Human observation instead of an automated halt.

Trade-offs

Soak periods make rollouts slow, and a slow rollout means a longer window in which two versions coexist — which itself carries risk and requires every change to be backwards compatible for longer.

They also delay fixes: an urgent correction cannot wait four hours per stage, which is why an expedited path must exist with its own criteria rather than being improvised. A rollout process with no fast lane will have one improvised during an incident, which is worse than a designed one.

Interview question

"Your rollout has five stages and you want to ship today. Which soak periods would you shorten, which would you refuse to, and what would you monitor differently if you shortened them?"