Model Rollouts: Shadow, Canary, Rollback

A code deploy changes behaviour you specified; a model swap changes behaviour nobody fully specified. The new checkpoint scores two points higher on your eval suite, and also, invisibly: refuses a category of requests it used to answer, emits JSON with a trailing comma one time in two hundred, and runs 40% longer on summarisation prompts, blowing a downstream token budget. None of this is in the eval suite, because eval suites sample the behaviour surface and the surface is unbounded. Rollout machinery exists to catch what sampling missed, while real users are still mostly protected.

The staged pipeline

Offline gates come first and stay cheap: the eval suite, safety probes, and regression sets built from past incidents. They are necessary and insufficient; their job is to reject obviously bad candidates so the expensive stages see few (see eval-driven development).

Shadow traffic runs the candidate on a mirror of live requests while users continue to see the incumbent. Nothing the candidate produces reaches anyone, so it is free of user risk and can run for days, accumulating the distribution offline evals cannot fake: real phrasing, real corpora, real tool schemas, real multi-turn mess. Comparison is the hard part, since there is no ground truth: teams diff refusal rates, output lengths, tool-call validity, format-parse success, and latency, then put an LLM judge on paired outputs for a quality verdict. Shadowing doubles inference cost on the mirrored slice, so it is usually a sampled percentage, stratified to overweight segments that matter (paying tenants, each supported language, each tool).

Canary exposes the candidate to a small share of live traffic, 1 to 5%, with automatic revert wired to guardrail metrics: error and refusal rates, TTFT and TPOT, parse failures, thumbs-down rate. Canary is what catches the interactive effects shadowing structurally cannot, because shadow users never respond to the new model; a canary user who rephrases after a confusing answer generates the follow-up turn the shadow never saw. The SRE playbook for sizing and judging canaries transfers largely intact (Google SRE Workbook, Canarying Releases); what changes is the metric set.

Rollback is a versioning discipline

Rolling back a model only works if "the model" names one immutable thing. Serve pinned snapshots, never mutable aliases; an upstream alias that silently re-points is a rollout you did not schedule and cannot revert. And prompts are tuned against a model's quirks, so a prompt that flourishes on the new model may regress on the old one: the deployable, revertible unit is the bundle of model snapshot, prompt version, decoding parameters, and tool schemas. Teams that version these separately discover during an incident that no combination on the shelf is known-good.

Two serving-layer couplings bite here too. Prompt caches key on the model version, so a canary split runs at a lower cache hit rate than either pure fleet, which both costs money and muddies latency comparisons. And nondeterminism muddies output diffs: identical requests produce different bytes across runs because dynamic batching changes kernel reduction order, so naive golden-output testing fails even at temperature zero (Thinking Machines Lab, 2025, Defeating Nondeterminism in LLM Inference). Compare distributions, not transcripts.

When it breaks

• The judge prefers its relatives. LLM judges systematically favour outputs stylistically close to their own family; a judge from the candidate's lineage inflates the candidate. Use a third-family judge, randomise pair order, and spot-check with humans.
• Canary percentages hide rare regressions. A failure at one-in-ten-thousand needs on the order of hundreds of thousands of canary requests to surface at all; slice guardrail metrics by segment, or the average buries the minority it broke.
• Shadow costs shape shadow samples. Under budget pressure the mirrored slice quietly shrinks or drops the long-context requests that cost most, which are exactly where models diverge hardest.
• Feedback contamination outlives the rollback. If the candidate's outputs were logged into training data, caches, or user-visible history before revert, the incident persists after the traffic switch; quarantine canary outputs until the release is blessed.