Traffic-Mix Skew

A canary takes 5% of traffic and its error rate is higher than the baseline from the first minute. The build is rolled back. The next release raises the alert threshold, and the release after that raises it again, until the canary can no longer fail.

Nothing was wrong with the code. The load balancer routed new connections to the canary, and new connections are not a random sample of traffic: they come from cold application starts, clients returning after a network change, crawlers and health checks, while long-lived keep-alive and WebSocket clients stayed on the baseline.

Why it matters

Canary analysis is a comparison of two populations, and the routing mechanism chose them. Where the choice correlates with behaviour, every conclusion is confounded, in both directions: false alarms that erode trust in the gate, and blindness to regressions in the traffic the canary never sees — a per-connection memory leak, a bug in a warm-session code path.

The second direction is the dangerous one, because it is silent. A skewed canary passes for the same reason it fails: the comparison never meant anything.

Implementation patterns

• Assign at the application layer by a stable hash of user or tenant id, so both arms draw from the same distribution and long-lived clients participate.
• Run an A/A comparison first. Route 5% to an identical build and measure the difference. That is your noise floor, and every threshold belongs above it.
• Stratify the comparison by route, client type and tenant class instead of comparing global aggregates, which is where a population shift hides inside an average.
• Size the window honestly. At a 1% base error rate, separating 1.0% from 1.1% takes on the order of hundreds of thousands of requests per arm; a 10-minute bake on a modest service does not have them.
• Guardrail on business outcomes — checkout completions, message send success — because the worst regressions return correct-looking responses.

Industry example

The failure mode is generic to progressive delivery and shows up wherever routing is done at the connection or edge layer rather than per request: connection-level canaries, region-scoped rollouts compared against a global baseline, and "new mobile version only" cohorts compared with everyone else. Mature deployment platforms in production address it the same way — request-level assignment, an automated analysis window with a pre-computed noise floor, and automatic abort on guardrail metrics rather than on raw infrastructure signals.

Failure scenarios

• Threshold inflation, where repeated false alarms train the team to widen the gate until it never fires.
• The invisible long-session regression, never routed to the canary at all.
• Region-correlated rollouts compared against a global baseline with a different traffic mix by time of day.
• Simpson's paradox: the aggregate improves while every segment worsens, because the mix changed.
• A canary that is too small to detect anything, presented as evidence of safety.

Trade-offs

Request-level assignment costs application work and a consistent identity to hash, and stratified analysis costs more metrics and more storage. The alternative costs either false confidence or a gate nobody trusts. The proportionate position for a low-traffic service is to skip canary statistics altogether and rely on a staged rollout with a rehearsed 5-minute rollback, stating plainly that the control is reversal speed rather than analysis.

When not to use it

When traffic is too small for any comparison to reach significance, do not build the analysis. A service at a handful of requests per second cannot produce a canary verdict in any useful window, and a dashboard that implies otherwise is worse than no dashboard. Spend the effort on rollback speed, feature flags and a real error budget instead.

Interview question

Q: Your canary system has been quiet for six months and everyone is pleased with it. What would you do to find out whether it can actually detect a regression?

What a strong answer covers: an A/A run to measure the noise floor · injecting a known regression behind a flag at a known magnitude and checking the gate fires · verifying how traffic is assigned and whether long-lived connections participate · computing the minimum detectable effect for the current bake window · and checking whether thresholds have been widened over time, which the change history will show.

Quick check

Quiz: Your canary receives only new connections. Name one false alarm and one blind spot. False alarm: more cold-cache and auth traffic raises errors and p99. Blind spot: per-connection leaks in long-lived clients never reach it.

Flashcard: What is the first canary experiment to run? — An A/A test, because without the noise floor every threshold is a guess.