Point-in-Time Correctness

Building a training set means joining features to historical outcomes: for each past event, what were the inputs, and what happened? The naive join uses the feature's current value, because that is what a normal join returns.

That value reflects everything that happened since, including the outcome being predicted. A customer's "total orders" today includes the order the model is being trained to predict; an account's "risk score" incorporates the fraud that was later detected.

Point-in-time correctness means retrieving the value as of the event's timestamp, using only what was knowable then.

Why it matters

Leakage produces spectacular evaluation results and poor production performance, which is the most expensive failure mode in applied machine learning — because it is not detected by any of the usual checks. The model scores well on held-out data, the team ships it, and the live metrics are disappointing in a way nobody can attribute.

The gap between offline and online performance is the symptom, and it is usually blamed on distribution shift, serving skew or the model itself before anyone examines the temporal correctness of the training join.

It is also easy to get wrong by hand and difficult to detect by inspection. A join that looks entirely reasonable — feature table joined to events on a customer key — is leaking, and nothing about the SQL reveals it.

Implementation patterns

• Every feature value carries a validity timestamp, and every training example carries an event timestamp; the join selects the latest feature value strictly before the event.
• Account for availability lag, not only event time. A feature computed hourly was not available at the event's timestamp even if its event time precedes it — the correct boundary is when the value would have been readable by the serving system, which is stricter and is frequently missed.
• One feature definition generating both the training computation and the serving computation, so the temporal semantics are enforced in both rather than reimplemented.
• Backfill support, so a new feature's history can be reconstructed correctly rather than accumulated over months.
• Test for leakage explicitly: a model whose offline performance is far above any plausible baseline should be treated as suspect rather than celebrated.
• Compare offline and online distributions per feature, which surfaces both leakage and skew.
• Version feature definitions, since changing one invalidates existing training data and there must be a policy on retraining versus retention.

Industry example

Airbnb's Chronon was built for exactly this problem across many teams and models: a single feature definition producing both the point-in-time-correct training dataset and the low-latency serving path, so that skew and leakage are structurally prevented rather than avoided by discipline.

The motivation is consistently organisational. With one team and one model, careful engineers keep two implementations aligned. With many teams, features are reimplemented, temporal semantics differ between implementations, and nobody can state what a feature means — which is the point at which a store earns its substantial operational cost.

Failure scenarios

• Joining current feature values to historical events, the default behaviour of a normal join.
• Ignoring availability lag, using a value whose event time precedes the event but which was not computable until later.
• Aggregates computed over the full history rather than up to the event time.
• Labels derived from data that also feeds a feature, which leaks the outcome directly.
• Offline and online implementations with different window boundaries, producing skew alongside leakage.
• Spectacular evaluation results accepted uncritically, when they are the primary symptom.
• Feature definitions changed without versioning, silently invalidating every existing training set.
• Backfilled history computed with current logic, reintroducing leakage into the reconstruction.

Trade-offs

Point-in-time joins are substantially more expensive than ordinary joins — they are temporal range joins over large histories, and generating a training set becomes a heavyweight computation rather than a query.

Maintaining the required history also costs storage: every feature value's full timeline must be retained for as long as training sets may need to reach back, which is considerably more than serving requires.

And the machinery — a feature store, a definition language, two consistent storage tiers — is a real platform with a real operational burden, on the inference request path.

The trade is compute, storage and operational complexity in exchange for models whose offline performance predicts their online performance. For a single model maintained by the team that built it, careful discipline is proportionate. For an organisation with many models, the discipline does not survive contact with turnover and reimplementation, and the structural guarantee is what is being bought.

Interview question

"Our model scores 0.94 offline and performs no better than the baseline in production. Give me your three most likely hypotheses in order, tell me how you would test the first one, and tell me what in our pipeline would have prevented it."