Your canary analysis compares the canary's error rate against current production. Results are noisy and unreliable. What is methodologically wrong?

What the interviewer is testing

Whether you can spot a confounded comparison, which is the most common defect in canary analysis.

What is wrong

You are comparing a freshly deployed instance against long-running ones, so the change is confounded with everything else that differs between a new process and a warm one:

Cold caches. JIT compilation state — a JVM or similar runtime takes time to reach steady-state performance. Empty connection pools. Cold buffer pools in any local store. Different position in the load balancer's rotation.

All of these make a new instance slower and more error-prone regardless of the code, so the canary looks worse than baseline for reasons that have nothing to do with the change. Teams learn to discount the difference — and then discount a real regression.

The fix

Deploy a baseline alongside the canary. Take the current version, deploy a fresh set of instances of it at the same time, and compare canary against that baseline rather than against existing production.

Both populations are now equally cold, equally new, receiving equally distributed traffic. The only systematic difference is the change itself.

This costs one additional deployment and removes an entire category of false signal. It is what Netflix's automated canary analysis does, and it is why their comparisons can be trusted enough to gate promotion automatically.

The other methodological requirements

Sufficient volume for significance. At low traffic the comparison cannot distinguish a real regression from noise in any reasonable window, and promoting on an underpowered test is worse than no canary because it carries the authority of a measurement.

Representative traffic distribution. Routing by user hash can hand the canary all internal users, one region or a single large tenant.

Business metrics, not only infrastructure ones. Plenty of regressions are perfectly healthy from the system's point of view and quietly stop customers converting.

What a strong answer adds

Noting that the same confounding affects load testing and A/B analysis, and that the general remedy is the same: construct a control that differs only in the variable under test. Teams routinely compare against whatever was there before and attribute the difference to their change.

Common weak answers

Extending the canary duration, which reduces noise and does not remove the bias. Loosening the thresholds until it passes.