A recommendation team maintains separate feature computation for offline training and online serving, and skew between them is degrading model quality. How does a feature store with point-in-time correctness fix this, and what does it cost operationally?

What training/serving skew actually is

Two implementations of the same feature — one in the batch pipeline that produces training data, one in the service that computes it at inference time — drift apart. A boundary condition differs, a null is handled differently, a time window is defined inclusively in one and exclusively in the other, a currency conversion uses a different rate source.

The model was trained on one distribution and is served another, and the degradation is gradual, silent and extremely difficult to attribute — because both implementations are individually correct and both pass their own tests.

The second, subtler form is label leakage through time. The training set is built by joining features to historical outcomes, and if the feature values used are the current ones rather than the values as they were at the moment of the event, the model trains on information that did not exist yet. It performs spectacularly in evaluation and poorly in production, which is the most expensive failure mode in applied machine learning.

What a feature store provides

• One definition, two execution paths. The feature is defined once, and the store generates both the batch computation for training data and the streaming or on-demand computation for serving. Skew becomes structurally impossible rather than a matter of discipline.
• Point-in-time correct joins. Building a training set retrieves each feature's value as of the event's timestamp, not its current value — which is the mechanism that eliminates leakage and is the hardest thing to implement correctly by hand.
• An online store for low-latency reads and an offline store for training, kept consistent by construction.
• Backfill from historical data, so a new feature can be added and its history reconstructed without waiting months to accumulate.
• Reuse and discovery across teams, so a feature computed by one team is available to others with its semantics attached.
• Freshness monitoring per feature, since a stale feature degrades a model silently.

The operational cost, honestly

• It is a platform with its own failure modes, now on the inference request path. Its outage is a serving outage, and it needs the availability posture of a production dependency.
• Two storage systems to operate and keep consistent — online and offline — plus the pipelines between them.
• A definition language and its versioning: changing a feature definition invalidates existing training data, and there must be a policy for whether models are retrained or the old definition is retained.
• Backfill compute is substantial for a new feature over a long history.
• A genuine learning curve, and a team that adopts one without understanding point-in-time semantics gets the machinery without the guarantee.

When it is not worth it

With one model and one team, it is over-engineering. Two implementations kept consistent by one team that understands both is entirely manageable, and the store's value is in reuse across teams and enforcement across implementations — both of which require several teams to exist.

The trigger is organisational rather than technical: when features are being reimplemented by different teams, when nobody can say what a feature means, or when a leakage incident has already occurred. Airbnb's Chronon emerged from exactly that situation — many teams, many models, and the same features computed differently in each.