Batch & Streaming Pipelines
Event time versus processing time, watermarks, exactly-once semantics, backfills and orchestration.
5concepts
60flashcards
36minutes of reading
- 01 Backfills and Reprocessing Why recomputing history is a different operation from running a pipeline fast, the resource and correctness hazards specific to it, and the design choices that make a pipeline backfillable at all.
- 02 Event Time Versus Processing Time The two clocks every streaming system has, why using the wrong one produces results that change on replay, and the skew that makes correctness a latency tradeoff.
- 03 Orchestration, Dependencies and Idempotency What an orchestrator is actually for, why retries are the whole reason tasks must be idempotent, and the difference between scheduling on time and scheduling on data availability.