Training-Serving Skew

A model learns a relationship between features and outcomes. If the features it receives in production differ in definition, distribution or timing from those it was trained on, the learned relationship does not apply.

The failure is entirely silent. Every service is healthy, every request succeeds, and the model is simply worse than it was measured to be. Because the degradation is in quality rather than in availability, no alarm fires and the discrepancy is typically discovered through a business metric weeks later.

The causes

• Separate implementations of the same feature. The batch pipeline computes "average order value over 30 days" one way; the serving path computes it another. Both are reasonable; they differ.
• Time leakage. Training on data that would not have been available at prediction time — an outcome, a later aggregate, a field populated after the event. Offline metrics are excellent and the model cannot work.
• Distribution differences between the training snapshot and live traffic, particularly after a product change or a seasonal shift.
• Preprocessing divergence — normalisation, encoding, missing-value handling — implemented twice.
• Freshness differences. A feature computed hourly in training and in real time at serving, or vice versa.

The boundary that prevents it

A feature store as the seam. A feature is defined once and executed by both the batch path that produces training data and the online path that serves requests. When the definition exists in one place, divergence from separate implementations is impossible by construction rather than by discipline.

The corresponding data principle is one ingestion path, two consumption paths: events written once to a durable log, with the streaming path building low-latency features and the batch path building training data and long-window features. Duplicating ingestion produces two pipelines that disagree, which is the origin of the problem.

Implementation patterns

• Point-in-time correct training data, constructed so each row contains only what was available at that moment. This is the direct control for time leakage and it is genuinely fiddly to get right.
• Log the features actually used at serving, and train on those logged features rather than recomputing them. This eliminates the divergence entirely for the features it covers.
• Version models with their feature definitions and training data snapshot, so a regression can be attributed and a model reproduced.
• Monitor feature distributions in production against training, alerting on drift — the only detection mechanism that fires before the business metric does.

Industry example

Recommendation platforms operating at very large scale treat this as the dominant correctness concern, ahead of model architecture. The reasoning is that model improvements are incremental and skew is categorical: a skewed feature can erase the benefit of a year of modelling work, silently.

It is also why the serving path is kept deliberately narrow — lookup, retrieval and ranking only, with everything derived from historical events precomputed. A request budget of tens of milliseconds forbids computing anything over history, and the precomputation is exactly where the shared definition must live.

Failure scenarios

• Excellent offline metrics and disappointing production results, repeatedly, with no explanation.
• A feature that improves the model offline and does nothing online, because it leaked.
• A pipeline change that silently alters a feature definition for training but not for serving.
• Backfills computed with current code against historical data, producing features that could not have existed.

Trade-offs

A feature store is real infrastructure with its own operational burden, and for a small number of models with simple features it is over-engineering — a shared library holding the definitions may be sufficient.

Serving-time feature logging costs storage proportional to traffic and adds a dependency on the log's completeness. Point-in-time correctness makes training data construction substantially harder.

All of these are cheap relative to a class of failure that is invisible, expensive and easy to reintroduce.

Interview question

"Your model's offline evaluation improves by 8% and production metrics do not move. Give me four explanations, and tell me which one you would check first and how."