Champion-Challenger Contamination

A recommendation model is retrained monthly. The challenger is promoted automatically whenever it beats the champion on an offline metric computed from the last 30 days of logged interactions. It passes every month. Live engagement has not moved in a year.

The log is not a sample of user preference. It is a sample of user preference conditional on what the champion chose to show. Items the champion never surfaced have no interactions, so a challenger that would have surfaced them is scored as wrong. The metric, computed faithfully, rewards agreement with the champion.

Each promotion therefore produces a closer copy. The explored catalogue narrows, the evaluation set narrows with it, and nothing fails, so the arrangement can run indefinitely.

Why it matters

It is a governance failure that passes every technical check. The pipeline is correct, the metric is computed properly, the promotion criterion is documented and followed. The defect is in the arrangement, not in any component, which is exactly the class that model-risk review exists to catch and routinely does not, because reviews examine components.

It also has a commercial signature that appears long before anyone notices the engagement plateau: impression concentration rising on a shrinking set of items, with long-tail content becoming unrankable rather than merely badly ranked.

Implementation patterns

• Randomised exploration on a small share of traffic, logged and flagged, giving an unbiased evaluation slice. A few percent is usually enough, and the cost is a known, bounded amount of engagement traded for a valid measurement.
• Inverse propensity weighting, correcting logged outcomes by the probability the champion had of showing each item — which requires those probabilities to have been recorded at serving time. If they were not, the correction is permanently unavailable and no later analysis recovers it.
• A live holdout population that never receives the new model, so comparison is against reality rather than against the log.
• Catalogue coverage and impression concentration as monitored metrics, because they move months before engagement does.
• A promotion criterion that includes a live test, not only an offline threshold. Automatic promotion on an offline metric alone is the specific mechanism that makes this self-sustaining.

Industry example

The problem is general to any system whose outputs determine its next training set, and the mitigations above — logged propensities, an exploration slice, off-policy estimators — are the standard published answers from the contextual-bandit and counterfactual-evaluation literature from 2010 onward. The reason it recurs in production is organisational rather than technical: exploration traffic has a visible short-term cost and the contamination has no visible cost at all, so exploration is the first thing cut when a quarter looks weak.

Failure scenarios

• A year of monthly promotions on an improving metric with flat live outcomes, attributed to market conditions.
• Serving-time propensities never logged, so the entire history is unusable for off-policy evaluation and the fix begins from zero.
• Exploration removed during a cost-reduction exercise, with the evaluation framework left in place and quietly invalidated.
• The same contamination in a fraud or credit model, where declined applicants generate no outcome data, so the model is evaluated only on the population it approved — the classic reject-inference problem, structurally identical.
• A challenger that genuinely is better being rejected, which is the silent opportunity cost nobody measures.

Trade-offs

Exploration costs engagement now to buy a valid measurement later, and the exchange rate is visible in one direction only. That asymmetry is the whole governance problem, and the practical answer is to fix the exploration budget as a policy — a stated percentage — rather than re-arguing it each quarter against whoever owns the short-term number.

Propensity weighting avoids the engagement cost and pays in variance: estimates become noisy exactly where the champion's probabilities were small, which is where the interesting differences are.

When not to use it

Where the model does not influence the data it is later trained on, this concern does not exist. A forecasting model over exogenous demand, a classifier over documents that arrive regardless, a model scoring transactions it does not select — none has the loop, and adding exploration traffic to any of them is pure cost.

The test is one question: does this model's output change what data we collect next? If yes, offline evaluation on logged data is contaminated by construction. If no, offline evaluation is sound and the apparatus above is governance theatre.

Interview question

Q: A challenger model beats the champion on your offline metric every month for a year, and live engagement is flat. Nothing in the pipeline is broken. Explain what is happening, what you would change, and what you would need to have logged a year ago to fix it cheaply.

What a strong answer covers: the log as preference conditional on the champion's choices, so the metric rewards imitation · the narrowing loop and why it has no corrective term · impression concentration and catalogue coverage as the early signals · an exploration slice with a fixed budget set as policy · propensity weighting and the fact that serving-time probabilities must already exist · a live holdout · the same structure in credit and fraud as reject inference · and the test for whether a given model has the loop at all.

Quick check

Quiz: Why does an offline metric computed on logged interactions reward imitation? Because the log only contains items the current model chose to show, so a challenger's better recommendations appear as misses.

Flashcard: What must be logged at serving time to fix this later? The probability the model had of showing each item — without those propensities, off-policy correction is permanently unavailable.