A canary deployment looks healthy on average latency while a small percentage of users experience severe failures. Which signals should gate the rollout?
Show the full answer Hide the answer
Why the average is misleading
Averages hide concentrated failures. A change breaking 3% of requests moves the mean modestly and the p99 severely — and if the affected users are a specific segment (one region, one device class, one client version), the aggregate can look entirely normal while those users are completely broken.
The signals that should gate
1. Percentiles, not means. p99 and p99.9, compared against the baseline population rather than against a fixed threshold. A comparison against the control group is what removes seasonality and traffic-shape noise.
2. Segmented comparison. By region, device class, client version, and — for a streaming platform — by stream size, because load and behaviour concentrate on the largest streams and their users are the ones who notice.
3. Error rates by class, not in aggregate. A new error type appearing at low volume is more informative than a small change in the total.
4. Business metrics, at the same priority as technical ones. Streams started, playback success, chat messages sent. These catch the failures where the system works correctly and produces the wrong result — which technical monitoring structurally cannot see.
5. User-signal metrics — retries, abandonment, reloads — which respond to degradation that produces no errors at all.
The mechanism
- Compare canary against a control group, not against yesterday. Same time, same conditions, different version.
- Automatic promotion and automatic rollback, gated on the comparison, with a minimum sample size before a verdict.
- A long enough soak at each stage to accumulate signal, particularly for a connection tier where the population grows slowly.
- Sticky assignment, so a user experiences one version consistently and is not switched mid-session.
The gap that produces this exact failure
No segmentation and no business metric. A rollout gated purely on aggregate technical signals will promote a change that breaks a minority completely — and the discovery comes from support rather than from the pipeline.