Progressive Delivery for Models

A model that passes every offline test can still fail in production, because offline evaluation uses a static dataset and production has live traffic, real feature pipelines, latency budgets and users who respond to the output. Progressive delivery is the practice of learning that in stages, each answering a different question, each with a bounded cost of being wrong.

The stages and what each answers

Shadow deployment. The new model receives a copy of production traffic and its outputs are logged and discarded. No user is affected. This answers questions about correctness and cost: does it run at production load, what is its latency distribution, does its output distribution match expectations, and how often does it disagree with the incumbent. It cannot answer whether it is better, because nothing acts on its predictions and no feedback is generated.

Canary. A small share of real traffic, typically 1 to 5 percent, receives the new model. Now the outputs act on the world, so operational metrics are real and early quality signals appear. Watch error rates and latency first, since those move fast, then the model's output distribution, then any business metric with a short enough feedback loop to be visible at that traffic share.

Progressive rollout. Traffic increases in steps with a bake period at each level. The purpose is to give slow-manifesting problems, memory leaks, cache pressure, feedback loops, time to appear at a traffic level where they are still recoverable.

A/B test. A statistically designed comparison, powered to detect a specified effect on a chosen metric, run for a predetermined duration. This is the only stage that answers whether the new model is better, and it requires enough traffic and enough time to reach the necessary power.

Shadow and canary are frequently conflated, and the distinction is worth holding: shadow validates that the model works, canary begins to validate that it helps.

What to watch, in order of speed

Error rate and latency respond within minutes. Prediction distribution shifts respond within hours and are the earliest quality signal available, since a model whose output distribution has moved is behaving differently whether or not the business metric has caught up. Business metrics respond in days, and the delay is why a rollout schedule cannot be driven by them alone.

When it breaks

Shadow mode is more expensive than it looks. Running both models doubles inference cost, and for a large model that is a real budget line rather than a rounding error. Sampling a fraction of traffic into shadow rather than duplicating all of it is usually sufficient and is often not done.

Canary populations are not random. Routing by user ID hash is fine; routing by region, by the first requests of the day, or by whichever pods received the new version, is not, and produces a comparison confounded with whatever the routing correlated with.

Feedback loops corrupt the comparison. A recommender whose canary arm changes what users see changes the training data for both arms, and a ranking change affects the inventory available to the control. Interference of this kind means the naive comparison is biased, and detecting it requires deliberately designed experiments rather than more traffic.

Slow metrics do not fit inside a rollout. Retention, churn and lifetime value take weeks. Rolling out over days means shipping on a proxy metric and monitoring the slow one afterwards, which should be an explicit decision recorded at the time rather than an omission discovered later.

Rollback must stay available through every stage. Once the previous model is unloaded, the rollout has passed a point of no return that nothing in the process announced. Keeping the incumbent warm until the new version has been at full traffic through a full business cycle is the conservative default.