Train-Serve Skew

Offline evaluation says the new ranker lifts click-through by 4%. The online experiment measures 0.3%, within noise. Nothing in the model changed between the two. In the training pipeline, session_length was a SQL aggregation over a completed session; in the serving path, it is computed by a service that returns elapsed time so far. Both are called session_length, both are floats, both look reasonable, and they are different features.

Train-serve skew is the difference between the value a feature takes during training and the value it takes at inference for the same logical entity. It is not model drift, which is the world changing. It is a defect, present on day one, and it is not visible in any offline metric because offline evaluation only ever sees the training-side computation.

The three ways it appears

Implementation skew. The training pipeline is a batch job in SQL or Spark; the serving path is application code in Python or Go. Two implementations of the same definition diverge on rounding, null handling, timezone, string normalisation, or the treatment of an empty list. Every one of those is a small difference on most rows and a large difference on some.

Time-travel skew. Training joins a feature table as it is now, so the value reflects everything up to the time the query ran. Serving sees only what was known at request time. Any aggregation with an open upper bound is affected. This is the same defect as temporal leakage seen from the serving side, and point-in-time correct joins, where the feature value is reconstructed as of each label's timestamp, are the standard defence.

Availability skew. A feature depends on an upstream service that is fast enough in batch and times out at the p99 in serving. The training data has it for every row; production has it for 97% of requests, and the imputation path is exercised far more often than anyone modelled.

What actually prevents it

The reliable structural answer is a single definition, executed once. A feature store's core promise is that the same transformation code produces both the offline training set and the online serving values, so the two cannot diverge by construction. Where a feature store is not in play, the equivalent is a shared library that both paths import, with the batch job and the online service calling identical code.

Where sharing code is impossible, logging is the fallback and it is a good one: log the feature vector actually used to make each prediction, and build the next training set from those logs rather than by recomputing from the warehouse. The training data is then, by definition, in the serving distribution. The cost is that the model can only learn from features that already exist in the serving path, so adding a new feature requires a logging period before it can be trained on.

Detecting it

Skew shows up as a distribution difference between training features and logged serving features, per feature: compare quantiles, means, null rates and category frequencies between the two populations. A population stability index or a simple two-sample test per feature, run continuously, catches most of it.

The stronger check is a shadow comparison. Score a sample of rows through both paths and compare outputs directly. Any nonzero disagreement rate on identical inputs is a defect; this is the difference between measuring a symptom and testing the property you actually want.

When it breaks

Skew and drift look the same on a dashboard. Both appear as a distribution shift between training and serving. They need different responses: skew is a bug to fix in code, drift is a signal to retrain. Distinguishing them requires knowing whether the definition changed or the world did, which a monitoring system cannot infer.

Retraining hides it. Retraining on freshly logged data makes the model consistent with whatever the serving path currently computes, including its bugs. The metric recovers, the defect is now baked into the training distribution, and nobody investigates. A recovered metric after retraining is not evidence that the problem is understood.

Logging changes the features you can build. Training from logged features guarantees consistency and constrains you to what is already logged. Adding a genuinely new feature needs a logging window first, which turns a one-day modelling change into a several-week cycle. This is a real cost of the safest approach and worth budgeting for explicitly.

Batch and streaming aggregations disagree by design. A daily batch count and a streaming windowed count over the same events differ at every boundary, on late-arriving events, and on retractions. Where both exist for the same feature, they are two features, and treating them as one is skew with extra steps.