An app with 4.2 million installs enters a phased release. The store's schedule is fixed - 1% of auto-updating users on day one then 2, 5, 10, 20, 50 and 100 percent over seven days - and you can pause it but never un-ship it. The new build crashes on a code path about 12% of sessions reach. Roughly how many crashing sessions on day one, and does that schedule actually protect you?
Show the full answer Hide the answer
The assumptions, stated
Four numbers do the work, and only one of them is a guess:
- 4.2M installs. Measured.
- Share with automatic updates on. This is the one you must look up rather than assume; call it 70%, so about 2.94M devices are eligible for the phased schedule. The other 30% get nothing until they open the store themselves.
- Day-one slice: 1% of the eligible population, so roughly 29,000 devices.
- Share of those that open the app on day one. For a daily-use app call it 40%, so about 12,000 sessions.
- Share of sessions that reach the broken path: 12% - the guess.
The arithmetic
12,000 sessions × 12% gives roughly 1,400 crashing sessions on day one, from roughly 1,400 distinct devices at most. Range: with auto-update share anywhere between 50% and 85% and day-one open rate between 25% and 60%, the plausible band is 400 to 2,900.
Now the number that decides whether the schedule helps. Crash-free sessions inside the day-one cohort is about 88%. Crash-free sessions across the whole install base that day is about 1,400 bad sessions against roughly 1.7M total, so it moves from a baseline of 99.80% to 99.72%.
What the number rules in or out
1,400 crash reports is overwhelming signal, and the fleet-wide metric barely moves. That is the whole lesson. A phased release gives you a cohort small enough to be cheap and large enough to be statistically loud, and a dashboard that averages over all versions converts that loud signal into an eighth of a percentage point of noise.
So the schedule protects you only if three things are true:
- Crash-free sessions is computed per build version, and the alert fires on the new build's own rate against the previous build's rate, not against an absolute threshold.
- The alert threshold is set for a cohort of 12,000 sessions, where a 1% real regression is visible within hours. At 100% rollout the same alert would need days of data to separate a 1% regression from weekday variation.
- Someone is watching on day one. The schedule compresses 1% to 100% into seven days, and days three to five move from 5% to 20% - the window where a missed signal becomes several hundred thousand devices.
Which assumption dominates the error
Not the 12%. The automatic-update share dominates, because it scales the whole calculation and teams routinely assume it is near 100%. Measure it from your own install base by comparing version adoption curves against the schedule: if day-one adoption is far below 1% of installs, your eligible population is much smaller than you think and your day-one cohort may be too small to see anything.
When this is the wrong answer
The arithmetic above is about detection, and the schedule's real constraint is that it has no reverse gear. Pausing stops new automatic installs; the devices that already updated keep the bad build until you ship a new one, which costs another review cycle plus another seven-day ramp. For a crash the phased release is adequate because detection is fast and the exposed population is small. For a defect that writes bad data - a sync bug that corrupts the local store, a payment field sent in the wrong unit - 1% of 2.94M devices is still 29,000 corrupted datasets, and no amount of staging fixes that after the fact.
That class of change needs a server-side flag evaluated before the code path runs, so the blast radius is controlled by a value you own rather than by a schedule the store owns. The staged rollout limits who gets your binary; only a flag limits what the binary does.