advanced
3 min answer
You own the system that rolls changes out to thirty regions. During an incident the commander asks how fast you can get a fix everywhere. Walk me through your answer.
Show the full answer Hide the answer
What the interviewer is testing
Whether you have a number, and whether you understand that it is a designed property rather than a measurement. "As fast as possible" is the wrong answer; so is any figure that does not distinguish the normal path from the emergency one.
The good answer has two numbers and the conditions under which each applies.
The clarifying questions that change the answer
- Is the fix a revert or a new change? A revert to a previously-deployed artefact can be much faster, because the target state has already run everywhere.
- Is the incident global or regional? A regional incident does not justify a global fast path, and using one turns a contained problem into an uncontained change.
- What is the failure mode if the fix is wrong? This is the question that sets the wave size, and it is the one nobody asks under pressure.
The arc of a strong answer
- State the normal path. For example: six waves, 30 minutes of bake between them, automated health gates comparing error rate and latency against the unchanged regions. Roughly three hours end to end. The bake time is the whole value - it is what converts a global outage into a single-region one.
- State the emergency path and its cost. Three waves, 5 minutes of bake, one named approver. Under 20 minutes. This is not "the same thing faster" - it is a deliberate trade of detection time for propagation speed, and the risk is that a bad fix reaches everywhere before the signal arrives.
- Say who may authorise it, because an emergency path available to everyone becomes the normal path, and then there is no safe path at all.
- Point out that speed of rollout is speed of failure. Fastly's June 2021 incident is the reference point: a valid customer configuration triggered a latent bug and about 85% of its network returned errors. The organisation detected it within about a minute and had 95% of the network recovered within 49 minutes. Fast propagation is a capability that cuts both ways, and the mitigation for it is fast detection and fast revert, not slower rollout.
- Give the number for revert separately, and make sure it is smaller than the number for rolling forward. If reverting is as slow as deploying, the emergency path has no floor.
Common weak answers
- "We can push globally in two minutes." A boast that describes a blast radius, not a capability.
- "Region by region takes four hours." A number with no bake time attached is a schedule, not a safety property - sequential regions with no health gate provide almost none of the safety they appear to.
- "We'd have to check." The number belongs in the runbook, verified by a drill, because the moment it is asked is the moment nobody has time to work it out.
What a strong answer adds
That the interesting number is neither of the two: it is time-to-detect, because rollout safety is bake time measured against detection time. Thirty minutes of bake with a signal that takes an hour to move provides nothing. The honest version of the runbook states both, and the improvement work is usually on detection rather than on the rollout machinery.