A canary release looks healthy on p50 latency but a small set of enterprise tenants sees timeouts. Which metrics and gates should have caught it?

Why the canary passed

Aggregate metrics hide minority failures by construction. If enterprise tenants are 2% of requests, their complete failure moves the overall error rate by 2% and the p50 not at all. A gate on aggregate p50 and error rate is mathematically incapable of detecting this, and no amount of tightening the threshold fixes it — tighten it far enough and it fails on normal variance instead.

Two compounding factors are typical: canary traffic is routed randomly, so a small tenant segment may barely appear in the canary at all; and enterprise tenants have different characteristics — larger payloads, more data, more complex permission evaluation, more relations to traverse — so the defect is triggered by a shape of request the canary rarely saw.

The metrics that should gate a rollout

• Tail latency, not the median. p99 and p99.9. The p50 is the least informative percentile for release safety, because a defect affecting a minority appears only in the tail.
• Segmented by tenant class: enterprise, mid-market, self-serve. Any segment degrading fails the gate, regardless of its share of traffic.
• Segmented by request shape — payload size buckets, result-set size, feature flags active. A timeout that only occurs above 10,000 rows is invisible in an aggregate and obvious in a bucket.
• Segmented by region, client version and device class, for the same reason.
• Per-tenant error rate for the largest tenants individually, because the top few customers may each be significant enough to warrant their own signal.
• Business-level indicators: completed checkouts, successful syncs, messages delivered. These catch failures that are invisible in technical metrics, such as a change that succeeds technically and returns wrong results.
• Comparison against the control group, not against a fixed threshold. The canary and the baseline receive simultaneous traffic, so environmental variation is cancelled out and a genuine regression is detectable at a much smaller magnitude.

Making the routing deliberate

Random canary routing gives poor coverage of minority segments. Better:

• Deliberate inclusion of representative tenants from each class in the canary, chosen in advance.
• Internal and volunteer tenants first, which is a ring rather than a percentage.
• Sticky assignment, so a tenant experiences one version consistently rather than alternating — which also makes their experience diagnosable.

Traces as the gate's diagnostic layer

When a gate trips, the question is immediately "why", and comparing traces between canary and baseline for the same operation answers it faster than anything else: which span grew, which dependency was called more times, where the additional latency accumulated. A gate that fails without a diagnostic path produces a rollback and no learning, and the same change is attempted again a week later.

The principle

Aggregate metrics answer "is the system healthy?" and rollout gates need to answer "is anyone worse off?" — which is a fundamentally different question requiring segmented, comparative measurement. Any gate built on aggregates is protecting the majority and, by construction, sacrificing the minority.