Temporal Join

Enriching a stream against reference data raises a question that looks like a detail and is a correctness decision: should the join use the reference value now, or the value at the time the event occurred?

For a live operational decision, now is usually right. For anything reported, reprocessed or used to train a model, as-of-event-time is right — and using the current value instead means the same input produces a different output on every rerun.

Why it is where the defects live

A non-temporal enrichment join is silently non-deterministic. It passes tests, produces plausible output, and diverges from the original result whenever it is replayed. That breaks the two properties reprocessing depends on:

• Determinism, without which reprocessing produces a different answer rather than a corrected one.
• Reproducibility, without which a reported figure cannot be explained after the fact.

In machine learning it is the classic source of label leakage: training features computed with the reference value as it is today encode information that did not exist when the event happened, so the model learns a relationship unavailable at serving time and performs worse in production than in evaluation.

Implementation patterns

• Versioned reference state with validity intervals, so a lookup by (key, event time) is possible at all. Reference data stored as current-value-only makes temporal joins impossible retrospectively — which is why this must be decided before the history is lost.
• Prefer stream-to-table enrichment over stream-to-stream where the shape allows: state is bounded by the reference data rather than by a window, and the semantics are far easier to reason about.
• Windowed stream-to-stream joins for genuinely correlated events, with the window size trading missed matches against state growth — a product decision, stated in product terms.
• Interval joins where the events are causally ordered and the delay is bounded.
• Unmatched events routed somewhere, always. Reconciling the unmatched set is what turns a lossy join into a correct system, and it is the single most commonly omitted piece.
• Deferring the join to batch, which is a legitimate choice when correctness matters more than freshness and the match may arrive hours later.

Industry example

Marketplace, travel and commerce platforms hit this whenever an order stream is enriched with pricing, tax or merchant attributes. Joining against current pricing means last quarter's orders re-enrich at today's prices when the pipeline is reprocessed, and the restated figures do not match what was reported. The fix is not a tuning change; it is retaining the price history and joining as of the order timestamp.

The same reasoning drives the point-in-time correctness requirement in feature stores, which exists precisely because the intuitive join is wrong in a way that does not fail loudly.

Failure scenarios

• Joining against current state for anything reprocessed, producing silent drift.
• Reference data with no history, which forecloses the correct join permanently.
• Unbounded join state, where the window is set generously and never revisited.
• Unmatched events dropped, converting a join into undetectable data loss.
• Windows chosen by tuning rather than by meaning, so the semantics change whenever someone optimises.

Trade-offs

Temporal joins cost more: versioned reference state is larger, the lookup is more expensive, and the pipeline carries history it would otherwise discard. For high-volume enrichment against slowly-changing data, the storage overhead is real.

The judgement is per-use: operational decisions genuinely want current values and should say so explicitly. Anything that will be reported, audited, replayed or trained on wants as-of semantics, and retrofitting them after the history is gone is not possible at all.

Interview question

"You reprocess three months of orders after fixing a bug, and the restated revenue does not match what was originally reported — even for orders the bug never touched. What is the most likely cause?"