The Leak in Every Training Set: Feature Stores, Point-in-Time Joins, and the Train-Serve Contract
A fraud model can score perfect recall offline and block nothing in production, because its training join looked a few hours into the future. Feature stores exist to enforce one contract: a training row may only see what the serving system could have seen at that instant. Here is the mechanism, the arithmetic of the leak, and when the machinery is not worth its cost.
In September 2017 Uber described Michelangelo, its machine learning platform, and the industry copied one detail. Uber had "approximately 10,000 features in Feature Store," shared across teams, and wanted to guarantee "that the same data generation/preparation process is used at both training time and prediction time" (Del Balso & Hermann, 2017). Engineers behind that post founded Tecton, which Databricks agreed to acquire in August 2025. A product category grew out of one requirement: the numbers a model trains on must be the numbers it will see when it serves.
That sounds like bookkeeping. It isn't, because getting it wrong makes offline metrics go up. Google's data validation team named one mechanism time travel, and removing a single training-serving skew in the Google Play recommender lifted the install rate on the store's main landing page by 2% (Breck et al., 2019, MLSys).
Why this matters: Every supervised model trained on event data joins labels to features across time, and the naive join leaks the future. Point-in-time correctness is the one property no test catches by accident, because violating it improves every offline metric. Feature stores, feature platforms and serving logs are three answers to the same contract.
TL;DR
- A naive join on entity and date lets a training row see values computed after its prediction time. The leak raises offline accuracy, so no gate flags it.
- The correct operation is an as-of join on availability time, not event time. For a nightly job landing at 03:00, an event-time join hands the model a fresher value than serving will have for 12.5% of rows.
- A batch feature is \(L + P/2\) old on average at serving: 15 hours for a daily job with a 3-hour lag, and up to 27.
- In the worked example, a naive join gives a fraud rule 4/4 offline recall and blocks $0 of $1,590 live. The point-in-time version reports ¾ and blocks $1,280.
- Leakage errors affected 294 papers across 17 fields (Kapoor & Narayanan), and Google's ML Test Score calls the skew check "perhaps the most important and least implemented" test.
- Practitioners disagree on the fix: Rules of ML says train on logged serving features; Airbnb's Chronon team says log-and-wait "takes months" and backfills instead. Each wins under different conditions.
At a Glance
flowchart LR
SRC["Event sources"] --> DEF["One feature definition"]
DEF --> OFF["Offline store, full history"]
DEF --> ON["Online store, latest values"]
LBL["Labels with prediction times"] --> PIT["As-of join on availability time"]
OFF --> PIT
PIT --> TRAIN["Training set and model"]
ON --> SERVE["Serving request"]
TRAIN --> SERVE
SERVE --> LOG["Logged feature vectors"]
LOG --> CHECK["Key-joined parity check"]
PIT --> CHECK
classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
class SRC,LBL blue
class DEF purple
class OFF,ON slate
class PIT amber
class TRAIN,SERVE teal
class LOG,CHECK emeraldEverything except the amber node is plumbing.
Before Feature Stores: Pipeline Jungles and Time Travel
The problem predates the product. In 2012 Kaufman, Rosset, Perlich and Stitelman formalised leakage, information about the target that would not legitimately be available at prediction time, and prescribed a "learn-predict separation" (Kaufman et al., 2012, TKDD 6(4)). Competitions expose leaks on a leaderboard; production has none.
In 2015 Sculley and colleagues described the organisational version: a mature system "might end up being (at most) 5% machine learning code and (at least) 95% glue code," with data preparation a jungle "of scrapes, joins, and sampling steps" (Sculley et al., 2015, NeurIPS). Their configuration-debt examples read like a feature store's requirements: "Feature B is not available on data before 10/7."
Zinkevich's Rules of Machine Learning drew the map. Rule #31 warns that "if you join data from a table at training and serving time, the data in the table may change." Rule #32 says reuse code between the pipelines. Rule #29 is the radical fix: "save the set of features used at serving time, and then pipe those features to a log to use them at training time," which gave the YouTube home page "significant quality improvements" (Zinkevich, Rules of ML).
timeline
title From leakage theory to feature platforms
2012 : Kaufman et al. formalise leakage in TKDD
2015 : Sculley et al. name pipeline jungles
: Dataflow Model separates event time from processing time
2017 : Uber Michelangelo shares about 10000 features
: ML Test Score calls skew monitoring least implemented
2019 : Gojek and Google Cloud open-source Feast
: TFX data validation names time travel skew
2020 : Tecton emerges from stealth
: DoorDash scales a Redis online store
2024 : Airbnb open-sources Chronon with Stripe
: Hopsworks feature store paper at SIGMOD
2025 : Databricks agrees to acquire TectonUber turned the rules into architecture. Batch features were loaded "from HDFS into Cassandra on a regular basis," Samza jobs produced near-real-time aggregates, and a DSL "implemented as sub-set of Scala" applied the same expressions at training and prediction time. Models reading Cassandra features typically answered at P95 under 10 ms, and the busiest served more than 250,000 predictions per second (Del Balso & Hermann, 2017).
[IMAGE: Two-lane diagram of Michelangelo, 2017: HDFS and Spark offline, Kafka, Samza and Cassandra online, one "Feature DSL" box feeding both. Caption: "Two execution paths, one definition."]
Feast followed in January 2019 (Google Cloud, 2019), Tecton out of stealth with a $20 million Series A in April 2020 (Miller, 2020), Chronon from Airbnb and Stripe in 2024 (Chronon repository). Databricks now pitches Tecton's "Sub-10 ms latency" as context for AI agents (Databricks, 2025).
How Point-in-Time Correctness Actually Works
Three clocks, not one
Every feature value carries three timestamps; most leaks use the wrong one. Event time \(s\) is when the underlying events happened; for "spend yesterday" it is the end of yesterday. Availability time \(a\) is when a serving system could first read the value: when the batch job finished writing or the stream processor emitted. Prediction time \(t\) is when the model would have scored the row. A label's timestamp must be this moment, not the moment the outcome became known.
The Dataflow Model made the event-time versus processing-time split standard in streaming (Akidau et al., 2015, PVLDB 8(12)). Feature stores need it one level up: the pipeline's processing time is the model's availability time.
The as-of join, stated precisely
Let \(F\) hold rows \((e, s, a, v)\): entity, event time, availability time, value. For a training row with entity \(e\) and prediction time \(t\):
and null if the set is empty. The constraint \(a \le t\) excludes anything serving could not yet read. The constraint \(t - s \le \tau\) is the time-to-live, which stops a value from passing as current long after its entity went quiet.
Most tools implement the weaker version, constraining only \(s \le t\). Feast will "scan backward in time from the entity dataframe timestamp up to a maximum of the TTL," relative to each row rather than to query time; in its documented example, with a 2-hour TTL, a row whose event "happened 11 hours after the feature row" gets nothing (Feast documentation). Databricks exposes timestamp_lookup_key plus lookback_window (Databricks documentation).
The availability constraint is what usually goes missing. As of September 2026, Feast offers it as an opt-in: filter_by_created_timestamp=True adds created_timestamp <= entity_timestamp "to keep backfilled values from leaking into training data," though "not all offline stores support this flag yet." The default still trusts event time.
How much the event-time shortcut leaks
A batch job runs every \(P\) hours and lands \(L\) hours after its window closes. The two joins disagree whenever a prediction falls between a window closing and its values landing, so under uniform traffic
For a daily job finishing at 03:00 that is \(3/24 = 12.5\%\) of training rows, each trained on a value one full period fresher than serving will ever supply. For slow features that is noise; for spend or failed logins during an incident, the leaked value is the signal being predicted.
Staleness is a distribution, not a setting
The same model gives the age of a batch feature at serving. With predictions uniform over the schedule, age is \(A = L + U\), \(U \sim \text{Uniform}(0, P)\), so
A nightly job with a 3-hour lag serves features 15 hours old on average and 27 at worst. An hourly job with a 10-minute lag averages 40 minutes. Whether that matters depends on how fast the signal decays, which makes "go streaming" a cost decision: a stream processor, checkpointed state and a second code path.
TTL interacts with staleness. If an entity's updates arrive as a Poisson process with rate \(\lambda\), the probability that none falls inside the TTL is
A merchant updating once every three days under a 2-day TTL returns null \(e^{-2/3} \approx 51\%\) of the time. That is correct, provided training sees the same nulls. A training set built without the TTL fills those rows with ancient values while serving sends nulls: skew.
[IMAGE: Three-day timeline with daily event times and 03:00 landings. A 02:14 prediction on day 3 points red to the day-2 row (event-time join) and green to the day-1 row (availability join); a shaded 00:00 to 03:00 band reads "12.5%". Caption: "Event time says the row exists. Availability time says serving cannot read it yet."]
Windows must end at the prediction, and hops set the error
"Transactions in the last hour" needs a window ending at each row's own \(t\). A calendar bucket from 02:00 to 03:00 hands a 02:14 prediction every transaction up to 02:59, which is the same leak in disguise.
Exact per-row windows are expensive, so platforms pre-aggregate "hops" and sum them at read time. Chronon's author describes a 1-hour window on 10-minute hops as "1 hour plus up to 10 minutes" (Simha, QCon SF 2023, via InfoQ). If the window covers \([t - W - \delta, t)\) with \(\delta\) uniform on \([0, h)\) and events arrive at steady rate \(r\), the expected over-count is \(rh/2\), a relative bias of
That bias is harmless when both paths compute the same approximation. Chronon promises backfills "point-in-time accurate such that consistency with online serving is guaranteed" (Chronon repository), and the operative word is consistency. An exact offline window would be more accurate and worse.
Late data and the bitemporal trap
A stream processor emits an aggregate once its watermark passes; later events take a late-firing path or are dropped (Akidau et al., 2015). The data lake eventually holds everything, so a backfill counts events serving never saw. If a fraction \(\ell\) arrives after the watermark, online counts run low by \(\ell\) relative to offline, worst during outages and traffic spikes, exactly when fraud models matter.
The fix is bitemporal. Each event carries event time and ingestion time, and a backfill for time \(t\) uses only events with both at or before \(t\):
Without ingestion time, a backfill cannot separate what serving knew from what the warehouse learned later. The TFX team's time-travel example has this shape: click counts queried at training time "appear higher compared to the serving data" because clicks kept accumulating after serving (Breck et al., 2019). Offline copies are more complete than what serving saw, so the model learns to lean on signal that thins out under stress.
What the join costs
A naive join on (entity, date) is a linear hash join. An as-of join partitions by entity, sorts and merges, \(O((n+m)\log(n+m))\), and per-label windows explode each row into its \(k\) events, \(O(n \cdot k)\). Hopsworks reported that "Spark's lack of AsOf Left joins" meant writing their own "union, exploding, and early stop sort-merge join operators" (de la Rúa Martínez et al., 2024, SIGMOD). The price of not leaking is a sort.
Seeing It in Motion
Michelangelo, Feast, Tecton, Hopsworks and Chronon share one shape. The Hopsworks paper notes that "all existing commercial and open-source feature stores follow this same dual-database architecture," because a single HTAP database "has not yet been shown to be capable" of serving both workloads (de la Rúa Martínez et al., 2024).
flowchart TB
subgraph Sources["Sources"]
EV["Event stream"]
DB["Database snapshots"]
REQ["Request payload"]
end
subgraph Compute["Feature compute"]
REG["Registry and definitions"]
BJ["Batch backfill jobs"]
SJ["Streaming aggregations"]
OD["On-demand transforms"]
end
subgraph Stores["Stores"]
OFF["Offline store, bitemporal"]
ON["Online key-value store, TTL"]
end
EV --> SJ
EV --> BJ
DB --> BJ
REG --> BJ
REG --> SJ
REG --> OD
BJ --> OFF
BJ --> ON
SJ --> ON
SJ --> OFF
REQ --> OD
ON --> OD
classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
class EV,DB,REQ blue
class REG amber
class BJ,SJ,OD purple
class OFF,ON slateThe edge that matters is streaming into the offline store. Logging streaming outputs with emission timestamps lets a training join reproduce what serving saw, gaps included; otherwise the only offline copy is a backfill, more complete than serving and therefore wrong. The sequence below follows one scoring request, one late event, and the backfill that later builds the training row.
sequenceDiagram
participant N as Card network
participant S as Stream processor
participant O as Online store
participant M as Fraud model
participant L as Feature log
participant B as Nightly backfill
N->>S: Txn at 02:19, amount 480
S->>O: txn_count_10m is 2
N->>M: Score txn at 02:19
M->>O: Read card features
O-->>M: count 2, spend_day 61
M->>L: Log vector with read time
N-->>S: A txn stamped 02:17 arrives at 02:40
Note over S: Past watermark, missing from live count
B->>B: Recompute from lake, count 3
Note over B,L: Disagrees with the log unless ingest time is filtered[IMAGE: Swimlanes from 02:10 to 02:40: an event stamped 02:17 arrives at 02:40 with a dashed connector back; the online count steps 0, 1, 2, the backfill 0, 1, 2, 3. Caption: "One late event is enough for the backfill to disagree with the model."]
By the Numbers
| Measure | Value | Context | Source |
|---|---|---|---|
| Shared features, Uber | about 10,000 | Michelangelo, 2017 | Del Balso & Hermann |
| Peak throughput, Uber | over 250,000 predictions/s; P95 under 10 ms | Busiest models; Cassandra-backed latency | Del Balso & Hermann |
| DoorDash store ranking | over 1M predictions/s | Needs tens of millions of feature reads/s | Khan & Hassan, 2020 |
| DoorDash Redis rework | 3x capacity, 38% lower Redis latency | Serialisation, hashing, compression | Khan & Hassan, 2020 |
| Google Play skew fix | +2% app install rate | Features in training, missing in serving logs | Breck et al., 2019 |
| Leakage in published science | 294 papers, 17 fields | Eight-type taxonomy | Kapoor & Narayanan, 2023 |
| ML Test Score | 28 tests; skew monitor "least implemented" | Google production rubric | Breck et al., 2017 |
| Hopsworks online p99, one vector | 15% of SageMaker, 11% of Vertex latency | Vendor-authored benchmark | de la Rúa Martínez et al., 2024 |
| Hopsworks PIT-join read, 10M rows | 11x, 10x, 17x throughput vs SageMaker, Vertex, Databricks | Vendor-authored benchmark | de la Rúa Martínez et al., 2024 |
| Tecton serving | sub-10 ms latency, sub-100 ms freshness, 99.99% uptime | Vendor claim, August 2025 | Databricks, 2025 |
Sources: Del Balso & Hermann, 2017; Khan & Hassan, 2020; Breck et al., 2019; Kapoor & Narayanan, 2023; Breck et al., 2017; de la Rúa Martínez et al., 2024; Databricks, 2025. Hopsworks benchmarks are vendor-run and unreproduced; Tecton figures are marketing claims. The Kapoor and Narayanan preprint reports 329 papers, the journal version 294.
Measured wins from fixing skew are modest percentages on huge bases: skew rarely breaks a model, it caps it. And every infrastructure number is an online-store number, the half that batch-scoring teams can skip.
[IMAGE: Log-scale bars: an illustrative internal model at hundreds of reads per second, Uber at 250,000 predictions per second, DoorDash at tens of millions of feature reads per second. Caption: "Online cost scales with predictions times features."]
A Concrete Example
All numbers here are illustrative, built to replay on paper.
The card. Card C-481 (UTC) spends $23 and $38 on 1 March, then legitimate L1 ($42 at 09:10) and L2 ($18 at 18:55) on 2 March. On 3 March a fraud burst hits: F1 $310 at 02:14, F2 $295 at 02:16, F3 $480 at 02:19, F4 $505 at 02:21. Four chargebacks post on 5 March.
The feature table. A nightly job computes card_daily per calendar day and lands at 03:00 the next morning:
| Row | Event time \(s\) | Available \(a\) | txn_count_day |
spend_day |
chargebacks_30d |
|---|---|---|---|---|---|
| R1 | 01 Mar 23:59 | 02 Mar 03:00 | 2 | $61 | 0 |
| R2 | 02 Mar 23:59 | 03 Mar 03:00 | 2 | $60 | 0 |
| R3 | 03 Mar 23:59 | 04 Mar 03:00 | 4 | $1,590 | 0 |
| R4 | 05 Mar 23:59 | 06 Mar 03:00 | 0 | $0 | 4 |
Step 1: same-day join. Match each transaction to its own date's row. F1 to F4 all get R3: spend_day = 1,590. The 02:14 transaction "knows" about $1,280 of spend that has not happened. L1 gets R2, which already includes L2 from almost ten hours later: a smaller leak of the same kind.
Step 2: latest-snapshot join. Join on card ID against today's table. Every C-481 row gets R4, chargebacks_30d = 4. Across 10,000 cards, chargebacks_30d > 0 marks nearly every card that ever had fraud: the label, renamed.
Step 3: event-time as-of join. For F1, the latest row with \(s \le t\) is R2. But R2 lands at 03:00, 46 minutes after F1; at 02:14 the online store still holds R1.
Step 4: availability-time as-of join. The latest row with \(a \le t\) is R1 for all six labelled rows (spend_day = 61), which matches what serving returned in the sequence diagram. Steps 3 and 4 differ by one dollar here; for a burst straddling midnight, by the whole burst.
Step 5: fit a one-split stump on spend_day over L1, L2, F1 to F4:
| Join | Fraud rows | Legit rows | Learned rule | Offline recall | Offline precision |
|---|---|---|---|---|---|
| Same-day | 1,590 | 60 | spend_day over $825 | 4/4 | 100% |
| Availability as-of | 61 | 61 | no useful split | 0/4 | n/a |
The naive join yields a perfect classifier, splitting at the midpoint \((60 + 1{,}590)/2 = 825\). The honest join shows spend_day says nothing about a burst in progress.
Step 6: serve. At 02:14, 02:16, 02:19 and 02:21 the online store returns spend_day = 61, and the $825 threshold never fires. Production recall 0/4; blocked $0 of $1,590. The offline report said 100%.
Step 7: the feature honesty forces. The team adds txn_count_10m, transactions in the 10 minutes strictly before \(t\): F1 = 0, F2 = 1, F3 = 2, F4 = 3, legit rows 0. The best stump is txn_count_10m ≥ 1, flagging F2 to F4 with no false positives; F1, the first transaction of the burst, is invisible to any backward-looking count. Offline recall ¾, production recall ¾, blocked $295 + $480 + $505 = $1,280.
Step 8: the alignment variant. Computed on a calendar bucket from 02:10 to 02:20, F1's count would be 3 (F1, F2, F3), and the leak returns. "Strictly before \(t\)" does the work.
The honest pipeline scores worse offline and saves $1,280 more in production. That inversion is why this bug survives code review.
[IMAGE: 2x2 grid of confusion matrices, rows "Same-day join" and "Availability join with txn_count_10m", columns "Offline" and "Production": 4 TP then 0 TP; 3 TP in both. Production dollar bars read $0 and $1,280. Caption: "Leakage moves the good numbers into the wrong matrix."]
Where It Breaks
The label timestamp is wrong
The commonest leak is not in the feature join. A chargeback label dated to its posting day, or churn dated to account closure, drags a flawless as-of join forward to the outcome. In Shankar et al.'s interviews with 18 ML engineers, leakage, "assuming during training that there is access to data that does not exist at serving time," was "typically discovered after the model was deployed" (Shankar et al., 2022, arXiv:2209.09125).
Backfills recompute history with today's knowledge
A backfill uses today's code and dimension tables, so a merchant recategorised last month acquires the new category for 2024 transactions. Sculley et al. describe the mirror image: "a silent update that corrects the signal will have sudden ramifications for the model" (Sculley et al., 2015).
Two definitions of TTL
Databricks documents that the lookback window "is applied during training and batch inference," while "during online inference, the latest feature value is always used, regardless of the lookback window" (Databricks documentation). A model trained with a 7-day lookback sees nulls for dormant entities; serving hands it their last value, however old. Enforce expiry in serving code, then verify the nulls match.
Log-and-wait versus backfill
Training on logged serving vectors makes parity definitional, but it also trains on the serving path's bugs, and only on features already being logged. Chronon's author puts the wait bluntly: log-and-wait "takes months," while a point-in-time backfill lets a team "train a model and immediately know if the z-score is useful or not, without having to wait for 6 months" (Simha, via InfoQ).
This is the real disagreement, and neither side is wrong. Rule #29 optimises parity for stable features; Chronon optimises iteration for a system still discovering them. A logging mistake costs calendar time; a backfill mistake costs a silent leak.
Parity checks that compare the wrong thing
The ML Test Score's Monitor 3 asks that "training and serving features compute the same values," tracking "the number of features that exhibit skew" (Breck et al., 2017). Comparing marginal distributions catches broken pipelines and misses leaks, since a leaked feature looks plausible. What catches Step 1 is comparing the same entity at the same timestamp, as TFDV does with "a key-join between corresponding batches of training and serving data followed by a feature wise comparison" (Breck et al., 2019).
The platform cost nobody budgeted
A feature store is two databases, a compute layer, a registry and an on-call rotation. DoorDash, needing "tens of millions of reads per second," benchmarked five key-value stores, chose Redis, then reworked hashing and serialisation to triple capacity (Khan & Hassan, 2020). None of it improves a model. A team scoring nightly needs the as-of join and can skip the rest.
[IMAGE: 2x2 matrix, x "Online inference under 100 ms" (no/yes), y "Teams sharing features" (few/many); quadrants "SQL as-of joins", "Online store for latency-bound features", "Registry plus offline store", "Full feature platform". Caption: "Point-in-time correctness belongs in every quadrant. A feature store, in one."]
Alternative Designs
| Design | How it works | Key advantage | Key limitation | Best when |
|---|---|---|---|---|
| Hand-written as-of joins | SQL or merge_asof with availability timestamps |
No new infrastructure | Every team reimplements windows and TTL | One to three models, batch scoring |
| Log-and-wait | Train on logged serving vectors | Parity by construction | Months per new feature; bugs become truth | Stable, high-traffic feature sets |
| Registry-centric store (Feast) | Declarative views over your warehouse, PIT retrieval, materialisation | Open source, bring your own stores | Streaming correctness stays your problem | Strong in-house data engineering |
| Compute-centric platform (Chronon, Tecton) | Declare aggregations once; platform generates backfill, stream and serving | Identical window semantics | Heavier platform; DSL bounds expressiveness | Many real-time aggregates, many teams |
| Lakehouse-native (Databricks) | Time-series feature tables with managed online sync | Governance and lineage included | Vendor coupling; online semantics can differ | Already standardised on the platform |
No design removes the as-of join; they differ in who writes it. Only compute-centric platforms own the windowed aggregation too, which is why only they can promise matching window semantics.
How It Is Used in Practice
DoorDash is the online-store case study: store ranking alone makes more than a million predictions per second with dozens of features each. At that read rate storage stopped being the limiting factor and CPU decided cost; compact serialisation with compression tripled capacity, though embeddings, "inherently high in entropy," did not compress (Khan & Hassan, 2020).
Airbnb and Stripe built Chronon around backfills, with a check to "ensure that backfill data for model training and evaluation is consistent with what is being observed in online serving" (Chronon repository). Its introductory talk cited a survey putting 80% of practitioner time into feature engineering and pipelines (Simha, via InfoQ), Sculley's glue-code figure a decade on.
The practices that recur: store availability or ingestion time on every row and join on it; log served vectors for a sample of traffic and key-join them against backfills daily; define TTL and nulls once for both paths.
Insights Worth Remembering
-
Leakage is the only bug that improves your metrics. Automated gates reject worse numbers, and point-in-time errors produce better ones. Only a key-joined comparison against served values catches them.
-
"As of" means as of availability. An event-time join is correct on paper and leaks for \(L/P\) of rows in practice, 12.5% for a nightly job landing at 03:00.
-
Identical imprecision beats divergent precision. An 8% over-count shared by both paths is harmless; an exact offline window beside an approximate online one teaches misplaced trust.
-
A feature store is two databases and a contract, and only the contract is mandatory. Batch-scored models need point-in-time joins, not an online store.
-
Log-and-wait and backfill trade different currencies. One pays in calendar time and inherited bugs, the other in compute and silent-leak risk. Mature systems run both and compare.
-
The label timestamp is a feature-store problem. A perfect as-of join anchored to when an outcome was known leaks the outcome.
Open Questions
Can one database serve both workloads? The Hopsworks team says no HTAP system has yet been shown to handle both online throughput and training-scale scans (de la Rúa Martínez et al., 2024). Whether lakehouse tables with caches close the gap is open; the benchmarks are vendor-run.
How much leakage is tolerable? Hour-bucketed timestamps leak by a bounded, derivable amount, but no general method turns that bound into expected production loss; teams measure it with shadow deployments.
Should backfills be bitemporal by default? Filtering on ingestion time reproduces what serving knew, and discards corrections that might improve the model. As of September 2026 Feast's opt-in flag is the most visible mainstream support for the first option.
Are offline agent evaluations leaking? Replaying historical tasks against today's tools and indexes is a latest-snapshot join. Leakage is measured in ML-based science (Kapoor & Narayanan, 2023), not in agent benchmarks; that many are optimistic for this reason is plausible and unproven.
Sources and Further Reading
- Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). "Leakage in Data Mining: Formulation, Detection, and Avoidance." ACM TKDD, 6(4), 15. doi:10.1145/2382577.2382579
- Sculley, D., Holt, G., Golovin, D., et al. (2015). "Hidden Technical Debt in Machine Learning Systems." NIPS 2015. Paper
- Akidau, T., Bradshaw, R., Chambers, C., et al. (2015). "The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing." PVLDB, 8(12), 1792-1803. doi:10.14778/2824032.2824076
- Zinkevich, M. Rules of Machine Learning: Best Practices for ML Engineering. Google for Developers. Guide
- Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017). "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction." IEEE Big Data 2017. Google Research
- Breck, E., Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2019). "Data Validation for Machine Learning." MLSys 2019. Paper
- Kapoor, S., & Narayanan, A. (2023). "Leakage and the Reproducibility Crisis in Machine-Learning-Based Science." Patterns, 4(9), 100804. Journal, arXiv:2207.07048
- Shankar, S., Garcia, R., Hellerstein, J. M., & Parameswaran, A. G. (2022). "Operationalizing Machine Learning: An Interview Study." arXiv:2209.09125
- de la Rúa Martínez, J., Buso, F., Kouzoupis, A., et al. (2024). "The Hopsworks Feature Store for Machine Learning." SIGMOD 2024 Companion. doi:10.1145/3626246.3653389
- Del Balso, M., & Hermann, J. (2017). "Meet Michelangelo: Uber's Machine Learning Platform." Uber Engineering. Post
- Sell, T., & Pienaar, W. (2019). "Introducing Feast: An Open Source Feature Store for Machine Learning." Google Cloud Blog. Post
- Feast. "Point-in-Time Joins." Documentation, accessed September 2026. Docs
- Databricks. "Point-in-time support using time series feature tables." Documentation, accessed September 2026. Docs
- Khan, A., & Hassan, Z. S. (2020). "Building a Gigascale ML Feature Store with Redis, Binary Serialization, String Hashing, and Compression." DoorDash Engineering. Post
- Airbnb. Chronon. GitHub repository, open-sourced 2024. airbnb/chronon
- Simha, N. (2023). "Chronon: Airbnb's End-to-End Feature Platform." QCon San Francisco; InfoQ. Talk
- Miller, R. (2020). "Tecton.ai Emerges from Stealth with $20M Series A to Build Machine Learning Platform." TechCrunch. Article
- Gupta, A., Parkhe, M., & Wendell, P. (2025). "Tecton is Joining Databricks to Power Real-Time Data for Personalized AI Agents." Databricks Blog. Post
Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.