A recommendation model is retrained monthly. The challenger is promoted automatically when it beats the champion on an offline metric computed from the last 30 days of logged interactions. It passes every month, and live engagement has been flat for a year. What is happening?

What happens, step by step

The logged interactions were produced by the champion. Users could only click what the champion chose to show them, so the log is not a sample of user preference — it is a sample of user preference conditional on the champion's choices.

An offline metric computed on that log rewards a challenger for agreeing with the champion. A model that would surface genuinely better items scores poorly, because the items it would have shown are absent from the log and its recommendations are scored as misses. The evaluation selects for imitation.

Each promoted challenger is therefore a slightly closer copy of the champion. The metric improves monthly, the catalogue's explored surface narrows, and live engagement does not move — which is exactly the observed pattern, and it can continue indefinitely because nothing fails.

Where it amplifies

The narrowing is self-reinforcing. Fewer items shown means fewer items logged, means an even more constrained evaluation set next month. Long-tail items become unrankable rather than badly ranked, and the effect shows up commercially as rising concentration of impressions on a shrinking set, long before it shows up in any engagement metric.

What the user sees

Nothing dramatic. Recommendations that are reasonable and repetitive. The complaint, when it arrives, is qualitative — "it keeps showing me the same things" — which no dashboard is measuring.

What stops it

• Randomised exploration on a small share of traffic, logged and flagged, giving an unbiased slice on which to evaluate. A few percent is usually enough, and its cost is a known, bounded amount of engagement traded for a valid evaluation.
• Inverse propensity weighting on the logged data, which corrects for the champion's selection probabilities — and requires that those probabilities were recorded at serving time. If they were not, the correction is unavailable and no amount of later analysis recovers it.
• A live holdout that never sees the new model, so the comparison is against reality rather than against the log.
• Monitoring impression concentration and catalogue coverage as first-class metrics, because they move months before engagement does.

What would have to be true for it to self-heal

Nothing. The loop has no corrective term: every component is behaving as specified. This is the defining property of the failure class — it is not a bug in any part, it is a property of the arrangement.

When this is the wrong thing to worry about

Where the model does not influence its own training data, the concern evaporates. A fraud model scoring transactions it does not select, a forecasting model over exogenous demand, a classifier over documents that arrive regardless — none of these have the loop.

The test is one question: does this model's output change what data we collect next? If yes, offline evaluation on logged data is contaminated by construction and needs exploration or propensity correction. If no, offline evaluation is sound and adding exploration traffic is pure cost.