Streaming Video Encoding & Packaging Pipeline · View 08 of 22 · Structure
Stack choices worth arguing with
- Spanner for job and chunk-task state: this is the one place in the system that needs transactional correctness across zones, and the alternative is a lease protocol we would have to write and prove.
- Spot managed instance groups for the batch fleet, with gVisor around the decoder — reclaimable compute is the economic premise, and an untrusted codec parser is the assumed attack.
- Shaka Packager rather than a managed packaging service, because common encryption from one elementary-stream set is the requirement and a per-minute managed price answers the cost question for us.
Why the on-demand pool is separate
- It is the only fleet with a latency SLO, so it is the only one carrying reserved headroom. Mixing the two would either give the batch fleet an availability requirement it does not need, or give a waiting viewer a Spot queue.
Deliberate omission
- Roughly thirteen edges are left off: every component writes the audit log and emits to BigQuery. Drawing them would obscure the eight that carry the flow.
