pattern

Stream Enrichment

Attaching reference data to a stream by lookup against a materialised table rather than by joining two unbounded streams.

Joining two streams requires buffering both sides for the window in which a match might arrive, which means state proportional to volume times window, and a decision about what to do when one side never appears.

Most real requirements are not that. They are "attach the customer's tier to each order event", where one side is a slowly changing reference set rather than a stream of comparable rate. Enrichment handles it by materialising the reference side as a keyed table — kept current by consuming its own changelog — and looking up per record.

State is then bounded by the reference data's cardinality rather than by throughput, and there is no window to reason about.

The subtlety is which version of the reference data to attach. Looking up the current value gives a result that changes if the pipeline is replayed after the reference data changed, which breaks reproducibility. Attaching the value as of the event's own timestamp is a temporal join, is more expensive, and is what correctness requires when the enriched field feeds a financial figure.