Term Kind Topic What it is
Changelog Stream concept Stream-Table Duality A stream whose records are keyed updates, so replaying it from the beginning reconstructs a table — the same information in the other of its two forms.
Checkpoint Interval metric Stream Processing Frameworks How often a stateful processor persists its state and offsets, which trades steady-state overhead against how much work is redone after a failure.
Consumer Lag metric Streaming SLOs How far behind the newest record a consumer is, expressed in time rather than in message count, and its rate of change.
Event Time Versus Processing Time concept Windowing The distinction between when something happened and when the system saw it, which determines whether results are reproducible.
Feature Staleness metric Feature Freshness The age of the feature values a model scores against, and the divergence between how they are computed at training time and at inference time.
Freshness Requirement concept Streaming vs Batch How stale data may be before the decision it supports degrades — the only question that justifies streaming over batch.
Idempotent Sink pattern Exactly-Once Semantics A destination where writing the same record twice has the same effect as writing it once, which is what makes end-to-end exactly-once achievable at all.
Key Skew concept Partition Keys & Ordering An uneven distribution of records across partitions, which caps throughput at the busiest partition regardless of how many exist.
Keyed State Size metric Stateful Stream Processing The total state a job holds per key across all keys, which governs memory, checkpoint duration and recovery time.
Poison Message concept Dead Letter Handling A record that a consumer cannot process and cannot skip, which halts its partition entirely until someone intervenes.
Replay Pipeline pattern Kappa vs Lambda A single processing path that produces both live and historical results by re-running the same code over retained input, replacing the two-path Lambda arrangement.
Reprocessing Window concept Backfill & Reprocessing How far back a pipeline can be replayed, set by the shortest retention anywhere along the path rather than by intent.
Retention Cost metric Streaming Cost The storage bill for keeping a log replayable, which is set by retention multiplied by throughput multiplied by the replication factor.
Schema Compatibility Mode concept Streaming Schema Evolution The rule stating which schema changes a registry will accept, which encodes whether producers or consumers are expected to upgrade first.
Serving Latency Budget concept Real-Time Serving Layer The end-to-end time from an event occurring to its effect being queryable, allocated across ingest, processing and serving.
Stream Enrichment pattern Streaming Joins Attaching reference data to a stream by lookup against a materialised table rather than by joining two unbounded streams.
Sub-Second Aggregation Store tool Real-Time Analytical Stores A database built to ingest continuously and answer aggregate queries over recent data in milliseconds, occupying the gap between OLTP and the warehouse.
Transaction Log Stream concept CDC to Stream Change events derived from a database's write-ahead log — faithful to the table's mutations and to its internal model rather than to the business's.
Unbounded Dataset concept Streaming & Real-Time Data Input with no known end, which removes the option of waiting for completeness and forces every aggregate to be provisional.
Watermark Lag metric Watermarks & Late Data How far behind the newest event a watermark is held, which trades result latency against how much late data is included.