Automated Retraining and Its Triggers
When a retraining loop is worth building, the four trigger designs and what each optimises, and the failure modes that make automated retraining actively dangerous.
Automated retraining is presented as the endpoint of MLOps maturity. It is a tool with a specific applicability, and building it for a model that does not need it produces cost, risk and a pipeline nobody maintains. The prior question is whether the model's performance actually degrades on a timescale that manual retraining cannot handle.
The four triggers
Scheduled. Retrain every night, week or month. Simple, predictable, and it retrains whether or not anything changed, which wastes compute when the data is stable and is too slow when it is not. It is the right default for a model whose data has steady turnover.
Performance-triggered. Retrain when a monitored metric crosses a threshold. This is the most directly motivated trigger and it requires ground-truth labels with enough freshness to detect degradation before it matters, which many problems do not provide.
Drift-triggered. Retrain when the input or prediction distribution shifts beyond a threshold. Available without labels, which is why it is common, and it fires on shifts that do not affect performance and misses degradation without distributional change. It is a proxy and should be treated as one.
Data-volume-triggered. Retrain when enough new labelled data has accumulated to plausibly improve the model. This is the right trigger for problems where labels arrive slowly, and it is more principled than a schedule for the same situation.
Production systems typically combine a schedule as a floor with a drift or performance trigger for responsiveness.
What must be automated with it
A retraining trigger without automated validation is an automated way to deploy an unvalidated model. The loop needs the full gate: evaluation against a fixed benchmark, comparison against the incumbent with a significance requirement, behavioural regression suite, and an automatic hold if any of them fails. The default on failure should be to keep the current model and alert, never to deploy the new one and alert.
When it breaks
Feedback loops make the model train on its own output. A recommender's next training set is generated by what the current model showed, so retraining reinforces existing behaviour and narrows exposure over time. Breaking the loop requires deliberate exploration traffic whose data is retained and weighted, which is a cost paid continuously to keep the loop from closing.
Automated retraining automates data poisoning. If an attacker can influence the training data, an automatic pipeline incorporates their influence on a known schedule with no human inspection. Data validation before training is a security control in this setting, not a quality control.
Drift detection fires on things that do not matter. A new device type, a marketing campaign, a seasonal shift: all produce distribution change without performance loss. Retraining on each wastes compute and introduces risk for no benefit, which is why drift thresholds tend to be widened until the trigger stops firing at all.
Degradation without drift is invisible to the common trigger. If the relationship between features and label changes while the feature distribution does not, drift detection sees nothing. This is concept drift, it is the case where retraining is most needed, and the most widely deployed trigger cannot detect it.
Nobody watches a pipeline that works. A retraining loop that has run cleanly for a year is unmonitored in practice, and its failure is discovered by a downstream consumer. Alerting on the absence of a successful run, rather than only on failures, is what catches a silently stopped scheduler.
14 flashcards for this concept
Click a card to reveal the answer.