Streaming Join

Joining is trivial in batch, where both sides are complete and at rest. In streaming, the right-hand record may arrive before the left, after it, or an hour later, and the processor must decide how long to wait while holding state for everything unmatched.

Three shapes with quite different costs. A stream-table join enriches events against a materialised reference view — the most common and cheapest case, requiring state proportional to the table rather than the stream, though it raises a subtle correctness question about which version of the reference data an event should see. A stream-stream join requires buffering both sides within a time window, so state is proportional to window length times throughput, and the window is a direct trade of completeness against memory. A table-table join maintains a materialised result incrementally.

The failure everyone meets once: a stream-stream join with a generous window, high throughput and no retention policy, which accumulates state until the job dies — typically weeks after it was deployed, and always in production.

The design question worth asking before reaching for a streaming join at all is whether the enrichment can happen at the producer, where both pieces of information are already present. Very often it can, and the join disappears.