Platforms & Practice

The Leak in Every Training Set: Feature Stores, Point-in-Time Joins, and the Train-Serve Contract

A fraud model can score perfect recall offline and block nothing in production, because its training join looked a few hours into the future. Feature stores exist to enforce one contract: a training row may only see what the serving system could have seen at that instant. Here is the mechanism, the arithmetic of the leak, and when the machinery is not worth its cost.

In September 2017 Uber described Michelangelo, its machine learning platform, and the industry copied one detail. Uber had "approximately 10,000 features in Feature Store," shared across teams, and wanted to guarantee "that the same data generation/preparation process is used at both training time and prediction time" (Del Balso & Hermann, 2017). Engineers behind that post founded Tecton, which Databricks agreed to acquire in August 2025. A product category grew out of one requirement: the numbers a model trains on must be the numbers it will see when it serves.

That sounds like bookkeeping. It isn't, because getting it wrong makes offline metrics go up. Google's data validation team named one mechanism time travel, and removing a single training-serving skew in the Google Play recommender lifted the install rate on the store's main landing page by 2% (Breck et al., 2019, MLSys).

Why this matters: Every supervised model trained on event data joins labels to features across time, and the naive join leaks the future. Point-in-time correctness is the one property no test catches by accident, because violating it improves every offline metric. Feature stores, feature platforms and serving logs are three answers to the same contract.

TL;DR

  • A naive join on entity and date lets a training row see values computed after its prediction time. The leak raises offline accuracy, so no gate flags it.
  • The correct operation is an as-of join on availability time, not event time. For a nightly job landing at 03:00, an event-time join hands the model a fresher value than serving will have for 12.5% of rows.
  • A batch feature is \(L + P/2\) old on average at serving: 15 hours for a daily job with a 3-hour lag, and up to 27.
  • In the worked example, a naive join gives a fraud rule 4/4 offline recall and blocks $0 of $1,590 live. The point-in-time version reports ¾ and blocks $1,280.
  • Leakage errors affected 294 papers across 17 fields (Kapoor & Narayanan), and Google's ML Test Score calls the skew check "perhaps the most important and least implemented" test.
  • Practitioners disagree on the fix: Rules of ML says train on logged serving features; Airbnb's Chronon team says log-and-wait "takes months" and backfills instead. Each wins under different conditions.

At a Glance

flowchart LR
    SRC["Event sources"] --> DEF["One feature definition"]
    DEF --> OFF["Offline store, full history"]
    DEF --> ON["Online store, latest values"]
    LBL["Labels with prediction times"] --> PIT["As-of join on availability time"]
    OFF --> PIT
    PIT --> TRAIN["Training set and model"]
    ON --> SERVE["Serving request"]
    TRAIN --> SERVE
    SERVE --> LOG["Logged feature vectors"]
    LOG --> CHECK["Key-joined parity check"]
    PIT --> CHECK

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class SRC,LBL blue
    class DEF purple
    class OFF,ON slate
    class PIT amber
    class TRAIN,SERVE teal
    class LOG,CHECK emerald

Everything except the amber node is plumbing.

Before Feature Stores: Pipeline Jungles and Time Travel

The problem predates the product. In 2012 Kaufman, Rosset, Perlich and Stitelman formalised leakage, information about the target that would not legitimately be available at prediction time, and prescribed a "learn-predict separation" (Kaufman et al., 2012, TKDD 6(4)). Competitions expose leaks on a leaderboard; production has none.

In 2015 Sculley and colleagues described the organisational version: a mature system "might end up being (at most) 5% machine learning code and (at least) 95% glue code," with data preparation a jungle "of scrapes, joins, and sampling steps" (Sculley et al., 2015, NeurIPS). Their configuration-debt examples read like a feature store's requirements: "Feature B is not available on data before 10/7."

Zinkevich's Rules of Machine Learning drew the map. Rule #31 warns that "if you join data from a table at training and serving time, the data in the table may change." Rule #32 says reuse code between the pipelines. Rule #29 is the radical fix: "save the set of features used at serving time, and then pipe those features to a log to use them at training time," which gave the YouTube home page "significant quality improvements" (Zinkevich, Rules of ML).

timeline
    title From leakage theory to feature platforms
    2012 : Kaufman et al. formalise leakage in TKDD
    2015 : Sculley et al. name pipeline jungles
         : Dataflow Model separates event time from processing time
    2017 : Uber Michelangelo shares about 10000 features
         : ML Test Score calls skew monitoring least implemented
    2019 : Gojek and Google Cloud open-source Feast
         : TFX data validation names time travel skew
    2020 : Tecton emerges from stealth
         : DoorDash scales a Redis online store
    2024 : Airbnb open-sources Chronon with Stripe
         : Hopsworks feature store paper at SIGMOD
    2025 : Databricks agrees to acquire Tecton

Uber turned the rules into architecture. Batch features were loaded "from HDFS into Cassandra on a regular basis," Samza jobs produced near-real-time aggregates, and a DSL "implemented as sub-set of Scala" applied the same expressions at training and prediction time. Models reading Cassandra features typically answered at P95 under 10 ms, and the busiest served more than 250,000 predictions per second (Del Balso & Hermann, 2017).

[IMAGE: Two-lane diagram of Michelangelo, 2017: HDFS and Spark offline, Kafka, Samza and Cassandra online, one "Feature DSL" box feeding both. Caption: "Two execution paths, one definition."]

Feast followed in January 2019 (Google Cloud, 2019), Tecton out of stealth with a $20 million Series A in April 2020 (Miller, 2020), Chronon from Airbnb and Stripe in 2024 (Chronon repository). Databricks now pitches Tecton's "Sub-10 ms latency" as context for AI agents (Databricks, 2025).

How Point-in-Time Correctness Actually Works

Three clocks, not one

Every feature value carries three timestamps; most leaks use the wrong one. Event time \(s\) is when the underlying events happened; for "spend yesterday" it is the end of yesterday. Availability time \(a\) is when a serving system could first read the value: when the batch job finished writing or the stream processor emitted. Prediction time \(t\) is when the model would have scored the row. A label's timestamp must be this moment, not the moment the outcome became known.

The Dataflow Model made the event-time versus processing-time split standard in streaming (Akidau et al., 2015, PVLDB 8(12)). Feature stores need it one level up: the pipeline's processing time is the model's availability time.

The as-of join, stated precisely

Let \(F\) hold rows \((e, s, a, v)\): entity, event time, availability time, value. For a training row with entity \(e\) and prediction time \(t\):

\[ f(e, t) = v_{e, s^{\star}}, \qquad s^{\star} = \max\{\, s : (e, s, a, v) \in F,\ a \le t,\ t - s \le \tau \,\} \]

and null if the set is empty. The constraint \(a \le t\) excludes anything serving could not yet read. The constraint \(t - s \le \tau\) is the time-to-live, which stops a value from passing as current long after its entity went quiet.

Most tools implement the weaker version, constraining only \(s \le t\). Feast will "scan backward in time from the entity dataframe timestamp up to a maximum of the TTL," relative to each row rather than to query time; in its documented example, with a 2-hour TTL, a row whose event "happened 11 hours after the feature row" gets nothing (Feast documentation). Databricks exposes timestamp_lookup_key plus lookback_window (Databricks documentation).

The availability constraint is what usually goes missing. As of September 2026, Feast offers it as an opt-in: filter_by_created_timestamp=True adds created_timestamp <= entity_timestamp "to keep backfilled values from leaking into training data," though "not all offline stores support this flag yet." The default still trusts event time.

How much the event-time shortcut leaks

A batch job runs every \(P\) hours and lands \(L\) hours after its window closes. The two joins disagree whenever a prediction falls between a window closing and its values landing, so under uniform traffic

\[ \Pr[\text{event-time join} \ne \text{availability join}] = \frac{L}{P}. \]

For a daily job finishing at 03:00 that is \(3/24 = 12.5\%\) of training rows, each trained on a value one full period fresher than serving will ever supply. For slow features that is noise; for spend or failed logins during an incident, the leaked value is the signal being predicted.

Staleness is a distribution, not a setting

The same model gives the age of a batch feature at serving. With predictions uniform over the schedule, age is \(A = L + U\), \(U \sim \text{Uniform}(0, P)\), so

\[ \mathbb{E}[A] = L + \frac{P}{2}, \qquad \max A = L + P. \]

A nightly job with a 3-hour lag serves features 15 hours old on average and 27 at worst. An hourly job with a 10-minute lag averages 40 minutes. Whether that matters depends on how fast the signal decays, which makes "go streaming" a cost decision: a stream processor, checkpointed state and a second code path.

TTL interacts with staleness. If an entity's updates arrive as a Poisson process with rate \(\lambda\), the probability that none falls inside the TTL is

\[ \Pr[\text{null}] = e^{-\lambda \tau}. \]

A merchant updating once every three days under a 2-day TTL returns null \(e^{-2/3} \approx 51\%\) of the time. That is correct, provided training sees the same nulls. A training set built without the TTL fills those rows with ancient values while serving sends nulls: skew.

[IMAGE: Three-day timeline with daily event times and 03:00 landings. A 02:14 prediction on day 3 points red to the day-2 row (event-time join) and green to the day-1 row (availability join); a shaded 00:00 to 03:00 band reads "12.5%". Caption: "Event time says the row exists. Availability time says serving cannot read it yet."]

Windows must end at the prediction, and hops set the error

"Transactions in the last hour" needs a window ending at each row's own \(t\). A calendar bucket from 02:00 to 03:00 hands a 02:14 prediction every transaction up to 02:59, which is the same leak in disguise.

Exact per-row windows are expensive, so platforms pre-aggregate "hops" and sum them at read time. Chronon's author describes a 1-hour window on 10-minute hops as "1 hour plus up to 10 minutes" (Simha, QCon SF 2023, via InfoQ). If the window covers \([t - W - \delta, t)\) with \(\delta\) uniform on \([0, h)\) and events arrive at steady rate \(r\), the expected over-count is \(rh/2\), a relative bias of

\[ \frac{rh/2}{rW} = \frac{h}{2W} = \frac{10}{120} \approx 8.3\%. \]

That bias is harmless when both paths compute the same approximation. Chronon promises backfills "point-in-time accurate such that consistency with online serving is guaranteed" (Chronon repository), and the operative word is consistency. An exact offline window would be more accurate and worse.

Late data and the bitemporal trap

A stream processor emits an aggregate once its watermark passes; later events take a late-firing path or are dropped (Akidau et al., 2015). The data lake eventually holds everything, so a backfill counts events serving never saw. If a fraction \(\ell\) arrives after the watermark, online counts run low by \(\ell\) relative to offline, worst during outages and traffic spikes, exactly when fraud models matter.

The fix is bitemporal. Each event carries event time and ingestion time, and a backfill for time \(t\) uses only events with both at or before \(t\):

\[ \hat f(e, t) = g\big(\{\, x \in \mathcal{E}_e : \tau_{\text{event}}(x) \le t,\ \tau_{\text{ingest}}(x) \le t \,\}\big). \]

Without ingestion time, a backfill cannot separate what serving knew from what the warehouse learned later. The TFX team's time-travel example has this shape: click counts queried at training time "appear higher compared to the serving data" because clicks kept accumulating after serving (Breck et al., 2019). Offline copies are more complete than what serving saw, so the model learns to lean on signal that thins out under stress.

What the join costs

A naive join on (entity, date) is a linear hash join. An as-of join partitions by entity, sorts and merges, \(O((n+m)\log(n+m))\), and per-label windows explode each row into its \(k\) events, \(O(n \cdot k)\). Hopsworks reported that "Spark's lack of AsOf Left joins" meant writing their own "union, exploding, and early stop sort-merge join operators" (de la Rúa Martínez et al., 2024, SIGMOD). The price of not leaking is a sort.

Seeing It in Motion

Michelangelo, Feast, Tecton, Hopsworks and Chronon share one shape. The Hopsworks paper notes that "all existing commercial and open-source feature stores follow this same dual-database architecture," because a single HTAP database "has not yet been shown to be capable" of serving both workloads (de la Rúa Martínez et al., 2024).

flowchart TB
    subgraph Sources["Sources"]
        EV["Event stream"]
        DB["Database snapshots"]
        REQ["Request payload"]
    end
    subgraph Compute["Feature compute"]
        REG["Registry and definitions"]
        BJ["Batch backfill jobs"]
        SJ["Streaming aggregations"]
        OD["On-demand transforms"]
    end
    subgraph Stores["Stores"]
        OFF["Offline store, bitemporal"]
        ON["Online key-value store, TTL"]
    end
    EV --> SJ
    EV --> BJ
    DB --> BJ
    REG --> BJ
    REG --> SJ
    REG --> OD
    BJ --> OFF
    BJ --> ON
    SJ --> ON
    SJ --> OFF
    REQ --> OD
    ON --> OD

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff

    class EV,DB,REQ blue
    class REG amber
    class BJ,SJ,OD purple
    class OFF,ON slate

The edge that matters is streaming into the offline store. Logging streaming outputs with emission timestamps lets a training join reproduce what serving saw, gaps included; otherwise the only offline copy is a backfill, more complete than serving and therefore wrong. The sequence below follows one scoring request, one late event, and the backfill that later builds the training row.

sequenceDiagram
    participant N as Card network
    participant S as Stream processor
    participant O as Online store
    participant M as Fraud model
    participant L as Feature log
    participant B as Nightly backfill
    N->>S: Txn at 02:19, amount 480
    S->>O: txn_count_10m is 2
    N->>M: Score txn at 02:19
    M->>O: Read card features
    O-->>M: count 2, spend_day 61
    M->>L: Log vector with read time
    N-->>S: A txn stamped 02:17 arrives at 02:40
    Note over S: Past watermark, missing from live count
    B->>B: Recompute from lake, count 3
    Note over B,L: Disagrees with the log unless ingest time is filtered

[IMAGE: Swimlanes from 02:10 to 02:40: an event stamped 02:17 arrives at 02:40 with a dashed connector back; the online count steps 0, 1, 2, the backfill 0, 1, 2, 3. Caption: "One late event is enough for the backfill to disagree with the model."]

By the Numbers

Measure Value Context Source
Shared features, Uber about 10,000 Michelangelo, 2017 Del Balso & Hermann
Peak throughput, Uber over 250,000 predictions/s; P95 under 10 ms Busiest models; Cassandra-backed latency Del Balso & Hermann
DoorDash store ranking over 1M predictions/s Needs tens of millions of feature reads/s Khan & Hassan, 2020
DoorDash Redis rework 3x capacity, 38% lower Redis latency Serialisation, hashing, compression Khan & Hassan, 2020
Google Play skew fix +2% app install rate Features in training, missing in serving logs Breck et al., 2019
Leakage in published science 294 papers, 17 fields Eight-type taxonomy Kapoor & Narayanan, 2023
ML Test Score 28 tests; skew monitor "least implemented" Google production rubric Breck et al., 2017
Hopsworks online p99, one vector 15% of SageMaker, 11% of Vertex latency Vendor-authored benchmark de la Rúa Martínez et al., 2024
Hopsworks PIT-join read, 10M rows 11x, 10x, 17x throughput vs SageMaker, Vertex, Databricks Vendor-authored benchmark de la Rúa Martínez et al., 2024
Tecton serving sub-10 ms latency, sub-100 ms freshness, 99.99% uptime Vendor claim, August 2025 Databricks, 2025

Sources: Del Balso & Hermann, 2017; Khan & Hassan, 2020; Breck et al., 2019; Kapoor & Narayanan, 2023; Breck et al., 2017; de la Rúa Martínez et al., 2024; Databricks, 2025. Hopsworks benchmarks are vendor-run and unreproduced; Tecton figures are marketing claims. The Kapoor and Narayanan preprint reports 329 papers, the journal version 294.

Measured wins from fixing skew are modest percentages on huge bases: skew rarely breaks a model, it caps it. And every infrastructure number is an online-store number, the half that batch-scoring teams can skip.

[IMAGE: Log-scale bars: an illustrative internal model at hundreds of reads per second, Uber at 250,000 predictions per second, DoorDash at tens of millions of feature reads per second. Caption: "Online cost scales with predictions times features."]

A Concrete Example

All numbers here are illustrative, built to replay on paper.

The card. Card C-481 (UTC) spends $23 and $38 on 1 March, then legitimate L1 ($42 at 09:10) and L2 ($18 at 18:55) on 2 March. On 3 March a fraud burst hits: F1 $310 at 02:14, F2 $295 at 02:16, F3 $480 at 02:19, F4 $505 at 02:21. Four chargebacks post on 5 March.

The feature table. A nightly job computes card_daily per calendar day and lands at 03:00 the next morning:

Row Event time \(s\) Available \(a\) txn_count_day spend_day chargebacks_30d
R1 01 Mar 23:59 02 Mar 03:00 2 $61 0
R2 02 Mar 23:59 03 Mar 03:00 2 $60 0
R3 03 Mar 23:59 04 Mar 03:00 4 $1,590 0
R4 05 Mar 23:59 06 Mar 03:00 0 $0 4

Step 1: same-day join. Match each transaction to its own date's row. F1 to F4 all get R3: spend_day = 1,590. The 02:14 transaction "knows" about $1,280 of spend that has not happened. L1 gets R2, which already includes L2 from almost ten hours later: a smaller leak of the same kind.

Step 2: latest-snapshot join. Join on card ID against today's table. Every C-481 row gets R4, chargebacks_30d = 4. Across 10,000 cards, chargebacks_30d > 0 marks nearly every card that ever had fraud: the label, renamed.

Step 3: event-time as-of join. For F1, the latest row with \(s \le t\) is R2. But R2 lands at 03:00, 46 minutes after F1; at 02:14 the online store still holds R1.

Step 4: availability-time as-of join. The latest row with \(a \le t\) is R1 for all six labelled rows (spend_day = 61), which matches what serving returned in the sequence diagram. Steps 3 and 4 differ by one dollar here; for a burst straddling midnight, by the whole burst.

Step 5: fit a one-split stump on spend_day over L1, L2, F1 to F4:

Join Fraud rows Legit rows Learned rule Offline recall Offline precision
Same-day 1,590 60 spend_day over $825 4/4 100%
Availability as-of 61 61 no useful split 0/4 n/a

The naive join yields a perfect classifier, splitting at the midpoint \((60 + 1{,}590)/2 = 825\). The honest join shows spend_day says nothing about a burst in progress.

Step 6: serve. At 02:14, 02:16, 02:19 and 02:21 the online store returns spend_day = 61, and the $825 threshold never fires. Production recall 0/4; blocked $0 of $1,590. The offline report said 100%.

Step 7: the feature honesty forces. The team adds txn_count_10m, transactions in the 10 minutes strictly before \(t\): F1 = 0, F2 = 1, F3 = 2, F4 = 3, legit rows 0. The best stump is txn_count_10m ≥ 1, flagging F2 to F4 with no false positives; F1, the first transaction of the burst, is invisible to any backward-looking count. Offline recall ¾, production recall ¾, blocked $295 + $480 + $505 = $1,280.

Step 8: the alignment variant. Computed on a calendar bucket from 02:10 to 02:20, F1's count would be 3 (F1, F2, F3), and the leak returns. "Strictly before \(t\)" does the work.

The honest pipeline scores worse offline and saves $1,280 more in production. That inversion is why this bug survives code review.

[IMAGE: 2x2 grid of confusion matrices, rows "Same-day join" and "Availability join with txn_count_10m", columns "Offline" and "Production": 4 TP then 0 TP; 3 TP in both. Production dollar bars read $0 and $1,280. Caption: "Leakage moves the good numbers into the wrong matrix."]

Where It Breaks

The label timestamp is wrong

The commonest leak is not in the feature join. A chargeback label dated to its posting day, or churn dated to account closure, drags a flawless as-of join forward to the outcome. In Shankar et al.'s interviews with 18 ML engineers, leakage, "assuming during training that there is access to data that does not exist at serving time," was "typically discovered after the model was deployed" (Shankar et al., 2022, arXiv:2209.09125).

Backfills recompute history with today's knowledge

A backfill uses today's code and dimension tables, so a merchant recategorised last month acquires the new category for 2024 transactions. Sculley et al. describe the mirror image: "a silent update that corrects the signal will have sudden ramifications for the model" (Sculley et al., 2015).

Two definitions of TTL

Databricks documents that the lookback window "is applied during training and batch inference," while "during online inference, the latest feature value is always used, regardless of the lookback window" (Databricks documentation). A model trained with a 7-day lookback sees nulls for dormant entities; serving hands it their last value, however old. Enforce expiry in serving code, then verify the nulls match.

Log-and-wait versus backfill

Training on logged serving vectors makes parity definitional, but it also trains on the serving path's bugs, and only on features already being logged. Chronon's author puts the wait bluntly: log-and-wait "takes months," while a point-in-time backfill lets a team "train a model and immediately know if the z-score is useful or not, without having to wait for 6 months" (Simha, via InfoQ).

This is the real disagreement, and neither side is wrong. Rule #29 optimises parity for stable features; Chronon optimises iteration for a system still discovering them. A logging mistake costs calendar time; a backfill mistake costs a silent leak.

Parity checks that compare the wrong thing

The ML Test Score's Monitor 3 asks that "training and serving features compute the same values," tracking "the number of features that exhibit skew" (Breck et al., 2017). Comparing marginal distributions catches broken pipelines and misses leaks, since a leaked feature looks plausible. What catches Step 1 is comparing the same entity at the same timestamp, as TFDV does with "a key-join between corresponding batches of training and serving data followed by a feature wise comparison" (Breck et al., 2019).

The platform cost nobody budgeted

A feature store is two databases, a compute layer, a registry and an on-call rotation. DoorDash, needing "tens of millions of reads per second," benchmarked five key-value stores, chose Redis, then reworked hashing and serialisation to triple capacity (Khan & Hassan, 2020). None of it improves a model. A team scoring nightly needs the as-of join and can skip the rest.

[IMAGE: 2x2 matrix, x "Online inference under 100 ms" (no/yes), y "Teams sharing features" (few/many); quadrants "SQL as-of joins", "Online store for latency-bound features", "Registry plus offline store", "Full feature platform". Caption: "Point-in-time correctness belongs in every quadrant. A feature store, in one."]

Alternative Designs

Design How it works Key advantage Key limitation Best when
Hand-written as-of joins SQL or merge_asof with availability timestamps No new infrastructure Every team reimplements windows and TTL One to three models, batch scoring
Log-and-wait Train on logged serving vectors Parity by construction Months per new feature; bugs become truth Stable, high-traffic feature sets
Registry-centric store (Feast) Declarative views over your warehouse, PIT retrieval, materialisation Open source, bring your own stores Streaming correctness stays your problem Strong in-house data engineering
Compute-centric platform (Chronon, Tecton) Declare aggregations once; platform generates backfill, stream and serving Identical window semantics Heavier platform; DSL bounds expressiveness Many real-time aggregates, many teams
Lakehouse-native (Databricks) Time-series feature tables with managed online sync Governance and lineage included Vendor coupling; online semantics can differ Already standardised on the platform

No design removes the as-of join; they differ in who writes it. Only compute-centric platforms own the windowed aggregation too, which is why only they can promise matching window semantics.

How It Is Used in Practice

DoorDash is the online-store case study: store ranking alone makes more than a million predictions per second with dozens of features each. At that read rate storage stopped being the limiting factor and CPU decided cost; compact serialisation with compression tripled capacity, though embeddings, "inherently high in entropy," did not compress (Khan & Hassan, 2020).

Airbnb and Stripe built Chronon around backfills, with a check to "ensure that backfill data for model training and evaluation is consistent with what is being observed in online serving" (Chronon repository). Its introductory talk cited a survey putting 80% of practitioner time into feature engineering and pipelines (Simha, via InfoQ), Sculley's glue-code figure a decade on.

The practices that recur: store availability or ingestion time on every row and join on it; log served vectors for a sample of traffic and key-join them against backfills daily; define TTL and nulls once for both paths.

Insights Worth Remembering

  1. Leakage is the only bug that improves your metrics. Automated gates reject worse numbers, and point-in-time errors produce better ones. Only a key-joined comparison against served values catches them.

  2. "As of" means as of availability. An event-time join is correct on paper and leaks for \(L/P\) of rows in practice, 12.5% for a nightly job landing at 03:00.

  3. Identical imprecision beats divergent precision. An 8% over-count shared by both paths is harmless; an exact offline window beside an approximate online one teaches misplaced trust.

  4. A feature store is two databases and a contract, and only the contract is mandatory. Batch-scored models need point-in-time joins, not an online store.

  5. Log-and-wait and backfill trade different currencies. One pays in calendar time and inherited bugs, the other in compute and silent-leak risk. Mature systems run both and compare.

  6. The label timestamp is a feature-store problem. A perfect as-of join anchored to when an outcome was known leaks the outcome.

Open Questions

Can one database serve both workloads? The Hopsworks team says no HTAP system has yet been shown to handle both online throughput and training-scale scans (de la Rúa Martínez et al., 2024). Whether lakehouse tables with caches close the gap is open; the benchmarks are vendor-run.

How much leakage is tolerable? Hour-bucketed timestamps leak by a bounded, derivable amount, but no general method turns that bound into expected production loss; teams measure it with shadow deployments.

Should backfills be bitemporal by default? Filtering on ingestion time reproduces what serving knew, and discards corrections that might improve the model. As of September 2026 Feast's opt-in flag is the most visible mainstream support for the first option.

Are offline agent evaluations leaking? Replaying historical tasks against today's tools and indexes is a latest-snapshot join. Leakage is measured in ML-based science (Kapoor & Narayanan, 2023), not in agent benchmarks; that many are optimistic for this reason is plausible and unproven.

Sources and Further Reading

  1. Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). "Leakage in Data Mining: Formulation, Detection, and Avoidance." ACM TKDD, 6(4), 15. doi:10.1145/2382577.2382579
  2. Sculley, D., Holt, G., Golovin, D., et al. (2015). "Hidden Technical Debt in Machine Learning Systems." NIPS 2015. Paper
  3. Akidau, T., Bradshaw, R., Chambers, C., et al. (2015). "The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing." PVLDB, 8(12), 1792-1803. doi:10.14778/2824032.2824076
  4. Zinkevich, M. Rules of Machine Learning: Best Practices for ML Engineering. Google for Developers. Guide
  5. Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017). "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction." IEEE Big Data 2017. Google Research
  6. Breck, E., Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2019). "Data Validation for Machine Learning." MLSys 2019. Paper
  7. Kapoor, S., & Narayanan, A. (2023). "Leakage and the Reproducibility Crisis in Machine-Learning-Based Science." Patterns, 4(9), 100804. Journal, arXiv:2207.07048
  8. Shankar, S., Garcia, R., Hellerstein, J. M., & Parameswaran, A. G. (2022). "Operationalizing Machine Learning: An Interview Study." arXiv:2209.09125
  9. de la Rúa Martínez, J., Buso, F., Kouzoupis, A., et al. (2024). "The Hopsworks Feature Store for Machine Learning." SIGMOD 2024 Companion. doi:10.1145/3626246.3653389
  10. Del Balso, M., & Hermann, J. (2017). "Meet Michelangelo: Uber's Machine Learning Platform." Uber Engineering. Post
  11. Sell, T., & Pienaar, W. (2019). "Introducing Feast: An Open Source Feature Store for Machine Learning." Google Cloud Blog. Post
  12. Feast. "Point-in-Time Joins." Documentation, accessed September 2026. Docs
  13. Databricks. "Point-in-time support using time series feature tables." Documentation, accessed September 2026. Docs
  14. Khan, A., & Hassan, Z. S. (2020). "Building a Gigascale ML Feature Store with Redis, Binary Serialization, String Hashing, and Compression." DoorDash Engineering. Post
  15. Airbnb. Chronon. GitHub repository, open-sourced 2024. airbnb/chronon
  16. Simha, N. (2023). "Chronon: Airbnb's End-to-End Feature Platform." QCon San Francisco; InfoQ. Talk
  17. Miller, R. (2020). "Tecton.ai Emerges from Stealth with $20M Series A to Build Machine Learning Platform." TechCrunch. Article
  18. Gupta, A., Parkhe, M., & Wendell, P. (2025). "Tecton is Joining Databricks to Power Real-Time Data for Personalized AI Agents." Databricks Blog. Post

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.