A canary takes 5% of traffic on a new build and is compared with the baseline on error rate and p99. What happens if the load balancer sends only new connections to the canary?

What happens, step by step

1. New connections are not a random sample of traffic. They come from cold app starts, clients returning after a network change, crawlers, health checks and anything that does not hold a keep-alive connection. Long-lived sessions — WebSockets, HTTP/2 connections from busy clients, connection-pooled server-to-server callers — stay on the baseline.
2. The canary therefore serves a different population: more authentication flows, more cold caches, more first-page requests, fewer deep session operations.
3. Its metrics diverge from the baseline immediately, and none of the divergence is caused by the code. Error rate is higher because auth failures concentrate there. p99 is worse because cold cache misses concentrate there.
4. The team either aborts a good release or, having seen a false alarm once, raises the thresholds until the canary can no longer fail. The second outcome is the dangerous one.

Meanwhile the opposite blind spot is open: a regression that only appears in long-lived connections — a memory leak per connection, a bug in a code path used after session state warms — cannot appear in the canary at all, because those clients never reach it.

Where it amplifies

Any assignment rule that correlates with behaviour has this shape: routing by region, by client version, by tenant size, or by "traffic from the new mobile release". Canary analysis is a comparison of two populations, and the deployment mechanism chose them.

What stops it

• Assign by a stable hash of user or tenant id, at the application layer, so both populations are drawn from the same distribution and long-lived clients participate.
• Run an A/A test first. Route 5% to the same build and measure the difference between two identical populations. That is your noise floor, and every threshold should be set above it.
• Stratify the comparison by route, client type and tenant class rather than comparing global aggregates, which is where Simpson's paradox lives.
• Size the window. At a 1% base error rate, distinguishing 1.0% from 1.1% needs on the order of hundreds of thousands of requests per arm; ten minutes at 5% of a modest service does not provide them, so the honest choices are a longer bake, a larger share, or a coarser detectable effect.
• Guardrail metrics that are business-visible — checkout completions, message send success — not only infrastructure signals, because the worst regressions are often silently correct responses.

What would have to be true for this to self-heal

Nothing. A biased canary passes and fails for reasons unrelated to the change, indefinitely, which is why the A/A run is not optional ceremony: it is the only evidence that the comparison means anything.

When this is over-engineering

For a service with a handful of requests per second, canary statistics cannot work at all — there is no sample. Prefer a staged rollout with fast rollback and an error budget, and be honest that the control is the speed of reversal rather than the analysis. The trade is explicit: you give up the chance to catch a regression before users meet it, and you buy the ability to end it in minutes. For most low-traffic services in production that is the better deal, because the alternative is a canary whose thresholds were tuned until it never fires.