Offline-Online Parity and Training-Serving Skew
Why the same feature computed by two pipelines is rarely the same number, the categories of divergence, and the architectural choices that eliminate rather than manage the problem.
Training reads a feature from a Spark job written in Scala over a data lake. Serving computes the same feature in a Java service from a Redis cache and a request payload. Two implementations, two languages, two data paths, one name. They agree until they do not, and when they diverge the model degrades without any error being raised anywhere.
This is training-serving skew, and it is consistently reported as one of the most common causes of a model performing worse in production than in evaluation.
Where the divergence comes from
Implementation drift. Two codebases computing the same logic diverge on edge cases: how nulls are handled, how strings are normalised, whether a division by zero yields null or zero, whether a timezone conversion happens. Each difference is small and each is silent.
Data source drift. The offline path reads a cleaned, deduplicated, late-arrival-corrected table. The online path reads a live stream or a request payload without those corrections. The feature is defined identically and the inputs are not the same data.
Time semantics. Offline aggregations use complete windows; online aggregations use whatever has arrived. A "count in the last hour" is exact offline and approximate online, and the approximation is biased low precisely when traffic is spiking.
Schema and encoding drift. A category added after training is unknown to the offline encoder's vocabulary and gets a default bucket online. A feature's units change upstream. Neither raises an error; both change what the model sees.
The architectural responses
Single definition, dual execution. Define each feature once in a declarative form and generate both the batch and the streaming implementation from it. This eliminates implementation drift by construction, which is the largest category, and it is the core design promise of a feature store. It does not address data source drift.
Log at serving, train on logs. Rather than recomputing historical features, log the exact feature vector used for every online prediction and use those logs as training data. Parity becomes definitional, since there is only one computation path. The costs are that a new feature cannot be trained on until it has been logged for long enough, and that the logging volume is substantial.
Continuous comparison. Compute features both ways for a sample of traffic and alert on distributional divergence per feature. This does not prevent skew and it detects it in hours instead of at the next model evaluation, which is usually the difference that matters.
When it breaks
Single definition does not survive optimisation. A feature defined once and executed by two engines will, sooner or later, have one path optimised for latency in a way that changes semantics: an approximate count-distinct online, a sampled aggregate, a cache with a stale entry. The definition still matches; the numbers do not.
Parity checks need the right statistic. Comparing means catches shifts and misses tail changes that matter most for a model's rare-but-important cases. Comparing full distributions per feature, and comparing the model's output distribution, catches more.
Skew is not always a bug. Sometimes the online value is right and the offline pipeline is wrong, for instance where the offline table has had late-arriving corrections applied that were genuinely unavailable at serving time. In that case the fix is to make training match serving, not the other way round, and teams reflexively assume the offline number is the truth.
The model can be retrained into the skew. Training on logged serving features makes the model consistent with whatever the online path does, including its errors. That is correct for parity and it silently accepts a bug as a specification, so logging-based training needs the online computation to be independently validated.
12 flashcards for this concept
Click a card to reveal the answer.