Windowing
Tumbling, sliding and session windows, and the aggregation each one answers.
4 to work through
-
intermediate
A live dashboard's "views in the last 30 minutes" reads about 30 times the true number. The job emits a 30-minute window advancing every minute; individual window values look correct when spot-checked; the sink is a time-series table and the panel sums the rows in the selected range. Lag and CPU are normal. What is happening and what do you change?
3 min answer -
advanced
Surge pricing must reflect supply and demand from seconds ago, apply consistently to everyone in an area, and never change mid-request. What is the architecture?
2 min answer -
advanced
Which windowing strategy fits which metric, and what is the property that breaks naive windowing?
2 min answer -
advanced
You implement session windows for user activity analysis. Downstream reports keep changing after publication. Explain.
2 min answer
4 terms in this topic
Event Time Versus Processing Time
The distinction between when something happened and when the system saw it, which determines whether results are reproducible.
conceptEvent Time vs Processing Time
The distinction between when something happened in the world and when the system got round to handling it.
metricWindow Overlap Factor
The ratio of a sliding window's size to its advance - the number of windows each record belongs to, and therefore the multiplier on state, output vol…
conceptWindowing Strategy
The choice of how a continuous stream is divided into finite groups for aggregation, which determines both the meaning of the result and the state it…
Neighbouring topics
Streaming & Real-Time Data
General material on continuous processing of unbounded data.
Streaming vs Batch
The freshness requirement that actually justifies streaming, and the cost of assuming one.
Exactly-Once Semantics
What the phrase really means, where it holds, and the idempotent sink underneath it.
Stream Processing Frameworks
Flink, Kafka Streams, Spark Structured Streaming — state, checkpointing and recovery.
Watermarks & Late Data
Deciding a window is complete when events can still arrive, and what to do when they do.
Stateful Stream Processing
Keyed state, state backends, checkpoint size, and the restore time that follows.
Stream-Table Duality
A changelog and a table as two views of the same thing, and materialising between them.
Kappa vs Lambda
One pipeline replayed versus two pipelines reconciled, and the maintenance each carries.
Streaming Schema Evolution
Changing an event's shape while a retained log still holds every older version of it.
Streaming Joins
Joining two unbounded streams, the buffering it needs, and the enrichment alternative.
Backfill & Reprocessing
Replaying history through changed logic without double-counting the live output.
CDC to Stream
Turning database changes into an event log, and how that differs from a domain event.
Real-Time Serving Layer
Where a low-latency read of a streaming aggregate actually lands.
Feature Freshness
How stale a feature can be before the model degrades, and the pipeline that follows.
Streaming SLOs
End-to-end latency, consumer lag and completeness as commitments rather than dashboards.
Partition Keys & Ordering
Ordering guaranteed only within a partition, and choosing the key that makes that enough.
Dead Letter Handling
The poison message that blocks a partition, and the queue nobody reads.
Streaming Cost
Always-on compute, retention and cross-zone traffic as the three bills that surprise.
Real-Time Analytical Stores
Druid, Pinot and ClickHouse — ingest-and-query engines for sub-second aggregation.