Batch & Streaming Pipelines
Event time versus processing time, watermarks, exactly-once semantics, backfills and orchestration.
7concepts
74flashcards
52minutes of reading
- 01 Backfills and Reprocessing Why recomputing history is a different operation from running a pipeline fast, the resource and correctness hazards specific to it, and the design choices that make a pipeline backfillable at all.
- 02 Change Data Capture and the Outbox Pattern Why writing to a database and a message broker from the same request cannot be made atomic, how log-based change data capture turns the database's own commit log into the event stream, and how the outbox pattern keeps that stream a contract rather than a leak of internal schema.
- 03 Event Time Versus Processing Time The two clocks every streaming system has, why using the wrong one produces results that change on replay, and the skew that makes correctness a latency tradeoff.
- 04 Orchestration, Dependencies and Idempotency What an orchestrator is actually for, why retries are the whole reason tasks must be idempotent, and the difference between scheduling on time and scheduling on data availability.
- 05 Stream Windowing: Tumbling, Sliding and Session How a windowing function assigns each event on an unbounded stream to finite groups, why the three standard shapes differ in state cost and in whether their boundaries are shared across keys, and how late data makes session windows merge after results have been emitted.
- 06 Exactly-Once Semantics, Precisely What "exactly once" actually guarantees, why it is a statement about effects rather than about deliveries, and the two mechanisms that provide it.
- 07 Watermarks and Allowed Lateness The heuristic that lets a streaming system decide a window is complete, why it is always wrong in one of two directions, and how triggers and lateness policies turn a single answer into a sequence of refinements.