Backfill & Reprocessing
Replaying history through changed logic without double-counting the live output.
6 to work through
-
beginner Multiple choice
A logic bug means the last 10 days of a derived table are wrong. An engineer proposes resetting the running consumer group's offsets back 10 days and letting the job reprocess. What does that actually do?
2 min answer -
advanced
A bug in a streaming job produced wrong results for three weeks. How should the backfill be designed so it does not cause a second incident?
3 min answer -
advanced
A bug is found in a stream processor that has been running for a month. How should the correction be handled?
2 min answer -
advanced
A logic bug means six months of a derived table are wrong. You must reprocess without disrupting live consumers. How?
2 min answer -
advanced
A transformation bug means months of derived data are wrong. What makes reprocessing feasible, and what breaks it?
2 min answer -
advanced
You reprocess six months of events through an updated job to fix a calculation error. Support is flooded with customer complaints. What went wrong?
2 min answer
2 terms in this topic
Replay Effect Suppression
Separating a stream processor's computation from the components that act on its output, so that reprocessing a month of data does not re-send a month…
conceptReprocessing Window
How far back a pipeline can be replayed, set by the shortest retention anywhere along the path rather than by intent.
Neighbouring topics
Streaming & Real-Time Data
General material on continuous processing of unbounded data.
Streaming vs Batch
The freshness requirement that actually justifies streaming, and the cost of assuming one.
Exactly-Once Semantics
What the phrase really means, where it holds, and the idempotent sink underneath it.
Stream Processing Frameworks
Flink, Kafka Streams, Spark Structured Streaming — state, checkpointing and recovery.
Windowing
Tumbling, sliding and session windows, and the aggregation each one answers.
Watermarks & Late Data
Deciding a window is complete when events can still arrive, and what to do when they do.
Stateful Stream Processing
Keyed state, state backends, checkpoint size, and the restore time that follows.
Stream-Table Duality
A changelog and a table as two views of the same thing, and materialising between them.
Kappa vs Lambda
One pipeline replayed versus two pipelines reconciled, and the maintenance each carries.
Streaming Schema Evolution
Changing an event's shape while a retained log still holds every older version of it.
Streaming Joins
Joining two unbounded streams, the buffering it needs, and the enrichment alternative.
CDC to Stream
Turning database changes into an event log, and how that differs from a domain event.
Real-Time Serving Layer
Where a low-latency read of a streaming aggregate actually lands.
Feature Freshness
How stale a feature can be before the model degrades, and the pipeline that follows.
Streaming SLOs
End-to-end latency, consumer lag and completeness as commitments rather than dashboards.
Partition Keys & Ordering
Ordering guaranteed only within a partition, and choosing the key that makes that enough.
Dead Letter Handling
The poison message that blocks a partition, and the queue nobody reads.
Streaming Cost
Always-on compute, retention and cross-zone traffic as the three bills that surprise.
Real-Time Analytical Stores
Druid, Pinot and ClickHouse — ingest-and-query engines for sub-second aggregation.