A pipeline must join two streams whose events arrive at different times. What makes streaming joins hard, and what are the options?

What makes them hard

Both sides are unbounded and neither is complete. A join needs matching records, and in a stream a match may arrive seconds or hours later — or never. So the join must hold state waiting for the other side, and that state must be bounded somehow.

The bounding mechanism determines the semantics, and the choice is a correctness decision rather than a tuning one.

The options

1. Windowed stream-to-stream join. Match events from both streams within a time window. State is bounded by the window; matches outside it are lost. Suited to genuinely correlated events — a request and its response, a click and its impression.

The window size is the trade: too small and legitimate matches are missed; too large and state grows.

2. Stream-to-table join (enrichment). One side is a stream, the other is a materialised table of current state — customer details, property attributes, reference data. The common case in practice, and it is cheaper because only the table's current state is held.

The correctness subtlety: should the join use the table's value now or its value when the event occurred? For enrichment used in a report, the second is usually correct and requires a temporal join against versioned state — which is where most defects live.

3. Interval join, where one side matches within a relative time range of the other. Asymmetric and useful for causally-ordered events.

4. Defer the join to a batch layer, where both sides are complete. Correct, and it forfeits the latency that motivated streaming — a legitimate choice when the join's correctness matters more than its freshness.

The design guidance

Prefer stream-to-table enrichment over stream-to-stream joins. It is cheaper, its state is bounded by the reference data rather than by a window, and its semantics are easier to reason about.

Where a stream-to-stream join is genuinely needed, make the window explicit in the product's terms — "we match a payment to an order within N minutes; beyond that it goes to reconciliation" — so the unmatched case is a designed path rather than silent loss.

Unmatched events must go somewhere, and reconciliation of the unmatched set is what turns a lossy join into a correct system.