Point-in-Time Correctness
The join that a feature store exists to get right, why a naive join on entity ID leaks the future into training labels, and what an as-of join costs to compute.
A fraud model scores a transaction at 14:32:07. To train it, you need the features as they were at 14:32:07: the customer's transaction count in the last hour, their average basket size, their account age. Joining the label table to the feature table on customer ID gives you the feature values as of whenever the feature table was last computed, which is after the fraud was discovered and the customer's statistics changed because of it. The model trains on evidence of the outcome it is predicting, reports excellent validation metrics, and fails in production.
The correct join
What is needed is an as-of join: for each training row with entity \(e\) and event timestamp \(t\), take the most recent feature value for \(e\) whose own timestamp is at or before \(t\).
Every feature carries its own valid-from timestamp, and every training row carries the timestamp at which a prediction would have been made. The join walks backwards from the label to the last feature value that existed then.
Implementing it means sorting both sides by entity and time and merging, which is a shuffle plus a sorted merge rather than a hash join. It is meaningfully more expensive than a naive join, and the expense is the point: it is the cost of not leaking.
Two timestamps, not one
Correctness needs both the event time of the feature and the time it became available to a serving system.
A feature computed from a daily batch job that runs at 03:00 and reflects data up to midnight is not available at 00:30, even though its event timestamp says midnight. Training on it teaches the model to use information that will not be there at inference. The gap between event time and availability time is the feature's lag, and honest training joins against availability time, not event time.
Where the batch job is late, the lag varies, which is why some systems record actual materialisation timestamps rather than assuming the schedule was met. This distinction is what separates a feature store that prevents leakage from one that merely organises features.
When it breaks
Label timestamps are frequently wrong. The label's timestamp must be the moment the prediction is made, not the moment the outcome is known. A churn label dated when the customer's account closed, joined as-of that date, uses features from the day of the outcome. This is the same leak in a different place, and it is the more common one.
Aggregation windows must end at the prediction time. A feature described as "transactions in the last 30 days" computed as a fixed calendar month includes transactions after the prediction point for most rows in that month. The window has to be relative to each row's timestamp, which is why point-in-time aggregation is expensive and why many teams approximate it and then wonder about the offline-online gap.
As-of joins are slow at scale, and the shortcuts leak. Bucketing timestamps to the nearest hour to enable a hash join, or precomputing daily snapshots, both reintroduce a bounded amount of leakage. Bounded leakage is a legitimate engineering choice and it must be a stated one, with the bound written down.
Backfilling a new feature is where correctness is usually lost. Adding a feature means computing its historical values, and doing that with today's code against today's data produces values that could not have existed at the time. The feature looks predictive in offline evaluation, ships, and does nothing.
No test catches this. Point-in-time errors produce better offline metrics, so every automated check passes. The only reliable detection is a comparison between offline feature values and the values actually logged at serving time for the same entities and timestamps, which is why serving-time feature logging is a requirement rather than an enhancement.
14 flashcards for this concept
Click a card to reveal the answer.