Feature Store  ·  View 14 of 21  ·  Runtime

Point-in-Time Training Set Assembly

Every value joined is the one whose ingestion timestamp was at or before the spine row's own timestamp. There is no second rule.

Editable source SVG draw.io All views
Request Entity–timestamp spine 500M rows Feature list 200 features Lookback 72 h default Pin Snapshot resolver Iceberg snapshot id Version resolver feature versions Join As-of join EMR Spark, p95 45 min Lookback filter reason-coded nulls Leakage detector ingest lag > horizon Emit Training set S3, immutable Manifest spine + versions + snapshot Consume SageMaker training Evaluation harness regenerate warning on dataset Point-in-Time Training Set Assembly External / third party Application we own Security / platform Data store batch failure / alternate Every value joined is the one whose ingestion timestamp was at or before the spine row's own timestamp. There is no other rule. v 1.0 · owner Data Platform Architecture · date 2026-09

Decisions

  • The three inputs — table snapshot, feature versions, lookback — are pinned before the join runs and recorded in the manifest, which is what makes a training set regenerable bit-identically.
  • A value outside the declared lookback returns a reason-coded null rather than reaching arbitrarily far back. Silent long-reach is how a stale value becomes a training label.
  • The leakage detector warns on the dataset rather than blocking it: a feature whose ingestion lag routinely exceeds the spine's horizon is a modelling problem, not a platform error.

Assumptions

  • 500M spine rows × 200 features p95 ≤ 45 min, hard ceiling 90 min; 10M × 50 p95 ≤ 5 min.
  • Default maximum lookback 72 hours, per-feature override.
  • Zero point-in-time violations tolerated — a value whose ingestion timestamp postdates its row timestamp is a Sev-1.

Risks

  • A late-arriving event changes future training sets and never restates a generated one. Two models trained a week apart on the same spine can therefore differ legitimately, and someone will read that as a bug.