You are handing a streaming pipeline to an operations team who have never run one. What must exist before they accept it?
Show the full answer Hide the answer
The signals they will be paged on
Consumer lag in time, not in messages — ten thousand messages means nothing without the rate. And alert on the derivative: growing lag means throughput is below arrival rate and will not recover, which is the page-worthy condition, often visible long before the absolute number looks alarming.
Per-partition lag, because a single partition falling behind on a hot key is invisible in the aggregate and is the common shape of a real incident.
Dead letter queue depth, with an owner. A DLQ nobody watches accumulates for months.
Checkpoint duration and state size, because those determine recovery time and they degrade slowly.
The runbooks, one per alert
Each with numbered steps, the exact commands, a verification step with a time bound, and an escalation point. Specifically:
- Lag growing — how to determine whether it is throughput, skew or a stuck consumer, and what may be scaled.
- A poison message — how to inspect the DLQ, and whether skipping is permitted for this pipeline (which is a decision that must be made in advance, not at 3am).
- A failed job — how to restart, expected restore time given state size, and what to do if it exceeds it.
- Backpressure — what is safe to do and what will make it worse.
The things that must be decided before handover, not during an incident
Whether records may be skipped. What the acceptable staleness is. Who owns the DLQ contents. Whether the pipeline may be stopped, and what happens downstream if it is.
The rehearsal
A game day covering at least: a consumer restart, a poison message, and a partition becoming hot. An operations team that has executed each once will handle them; one handed documentation will not.
What not to hand over
A pipeline whose recovery time nobody has measured. If restoring 200 GB of state takes forty minutes before it processes a single new record, operations need that number before they accept the rota.