Streaming Video Encoding & Packaging Pipeline  ·  View 14 of 22  ·  Runtime

Chunked Encode and Reclaim

Six stages across five lanes, with reclaim as a column rather than an exception.

Editable source SVG draw.io All views
Split Dispatch Encode Reclaim Stitch Verify Recipe plane Shot-boundary chunk plan Parameters per chunk Expected digest Control plane Materialise chunk tasks Lane capacity floor Pub/Sub dispatch Claim with lease Lease expiry detected Requeue one chunk All chunks present? Monotonic progress Encode fleet Spot worker starts Sandboxed decode Deterministic encode Preemption signal Stitcher joins chunks Boundary frames checked Storage Master range reads Chunk scratch, 48 h TTL Partial output discarded Elementary streams Scores recorded Cost Chunk size sets loss Spot, per-second billed ≤1 chunk-minute lost Cost per source hour Chunked Encode and Reclaim — What Each Layer Does The reclaim column is the architecture: it is a column, not an exception handler. v 1.0 · owner Media Platform Architecture

Chunk duration is the price of a reclaim

  • It is a tunable policy, not a constant: it sets the expected loss per withdrawal and the number of boundaries the stitcher has to make invisible.
  • Assumed 18% hourly reclaim at steady state and up to 60% of the fleet withdrawn inside a 10-minute window, sized so expected loss stays at or below one chunk-minute.
  • Whether to bound the loss this way or to checkpoint encoder state and resume is genuinely open (Question 6), and the answer depends on whether the chosen encoder can serialise its state at all.

Why determinism is load-bearing

  • Byte-identical output for a given source, recipe and build is what makes a duplicate execution indistinguishable from a single one — so at-least-once dispatch is sufficient and no distributed lock is needed on the hot path.
  • It is also what makes the lineage claim in view 12 true rather than aspirational.

Where the cost lane earns its place

  • Per-second billing plus Spot pricing is the premise of the whole economic argument; the cost lane is on the diagram because it is a design input, not a report.