Target Leakage

A churn model reports 0.97 AUC in cross-validation and 0.61 in production. Nobody made an arithmetic error, the validation split was stratified, the pipeline was tested. One feature was days_since_last_support_contact, populated by a workflow that fires when an account manager is told to save an account that has already decided to leave. The feature was available in the training warehouse and not available at the moment the prediction is needed.

Leakage is information in the training features that will not exist, in that form, at prediction time. It is the single most common reason a model that validated well fails on deployment, and it is invisible to every metric computed on the same data.

Why cross-validation cannot see it

Cross-validation estimates generalisation to new samples from the same distribution. A leaked feature is present in both training and held-out folds, correlates with the target in both, and the model uses it in both. Every fold agrees, the variance across folds is small, and the confidence in the number is entirely justified given the question it answers. The question it does not answer is whether the feature will be there, with the same meaning, at inference.

This is why leakage is caught by reasoning about time and provenance rather than by any statistical test. The one reliable signal is suspicion: performance that is much better than the problem should allow deserves an audit before it deserves a launch.

Three structural forms

Temporal leakage. A feature computed from data that arrives after the prediction point. total_lifetime_orders computed as of today, used to predict a purchase that happened six months ago, includes the purchase being predicted. The general rule is that every feature must be computed as of the prediction timestamp, and enforcing this is exactly what point-in-time correctness in a feature store means.

Train-test contamination. Any transform fitted on the full dataset before splitting: a scaler that saw the test set's mean, a target encoder whose category means include test rows, an imputer, a feature selector run on all the data, a SMOTE oversampling applied before the split so a synthetic point derived from a training row lands in test. All of these leak, and the fix is architectural: every fitted transform belongs inside the pipeline object that cross-validation refits per fold.

Proxy leakage. A feature that is a consequence of the label rather than a cause. A diagnosis code assigned after the outcome, an internal case status that only takes a certain value once the event has occurred, a row identifier correlated with class because the data were assembled class by class. This form is the hardest to detect automatically, because the feature looks ordinary and the correlation looks like signal.

Finding it

Sort features by importance and interrogate the top two or three, asking one question each: what process writes this field, and when does it fire relative to the prediction moment? Most leakage is found in that conversation rather than in the notebook.

Two mechanical checks help. A single feature that alone achieves near-target performance is nearly always leakage rather than an exceptional predictor. And a temporal holdout, training on everything before a date and testing after it, will show a gap against random cross-validation when temporal leakage is present, because the future information is no longer available in the same way.

When it breaks

The fix can be a feature you cannot afford to lose. Sometimes the leaking feature is genuinely available at prediction time for some rows and not others, and removing it costs real accuracy. The correct response is to model the availability explicitly, adding a "was this known yet" indicator and letting the model learn the conditional structure, rather than dropping or keeping it silently.

Group leakage is subtler than row leakage. Splitting a dataset with multiple rows per patient, user, or document randomly puts near-duplicate rows on both sides of the split. No individual feature leaks; the split does. Grouped cross-validation, splitting on the entity, is the fix, and it usually lowers the reported metric substantially, which is the point.

Some leakage is legitimate and should stay. If a feature is genuinely available at inference and simply happens to be highly predictive, it is not leakage. The test is availability and timing, not correlation strength. Removing strong features on suspicion alone is its own failure mode.

Leakage moves. A pipeline that was clean can start leaking when an upstream team changes when a field is populated. This is a monitoring problem, not a one-time audit: training-serving skew on feature distributions is the observable, and it needs to be watched continuously.