Stage Reliability Compounding
also called Serial Pipeline Availability, Chain On-Time Rate
The fact that a chain of dependent pipeline stages is on time only as often as the product of the stages' on-time rates, so excellent stages still produce an unreliable result.
A six-stage pipeline publishes the 07:00 revenue dashboard. Every stage meets its deadline 99% of days. The dashboard is late about 21 mornings a year. On-time probability for stages in series is the product of the individual probabilities (0.99^6 = 0.941), so a chain is always less reliable than its weakest link, and the gap widens with every stage added.
Why it matters
Pipeline owners are judged per stage and see green dashboards; consumers see only the end product. Without the arithmetic, a platform can have every component "meeting its SLO" and a product that misses its commitment about one day in seventeen. The compounding also explains why adding layers (silver, gold, platinum) quietly lowers freshness reliability even if each layer is well built.
Implementation patterns
- Calculate the chain: multiply per-stage on-time rates and compare with the product-level commitment before promising one.
- Remove stages that have no consumer; each deletion lifts the product more than tuning a good stage.
- Add slack: schedule stages with headroom so a 20-minute slip is absorbed rather than propagated.
- Partial publish: ship yesterday's numbers with a visible freshness stamp when a stage misses.
- Define the SLO at the product (for example, 95% of days by 07:00 and every day by 09:00), with an error budget the platform team spends on changes.
- Break shared fate: stages on one cluster or one orchestrator are correlated; isolate or monitor them as one.
Industry example
Google's SRE book makes the same argument for services: a system's availability is bounded by the product of its critical dependencies' availabilities, which is why it recommends that a service's own target be set well above what its dependencies allow it to promise. The mechanism is the same for scheduled stages: independent probabilities in series multiply, so six stages at 99% give 94.1%.
Failure scenarios
- Correlated misses: a cluster outage hits four stages at once, so the real distribution has long bad weeks and many perfect ones, which hides the true rate.
- Retries stacked: each stage retries for 30 minutes, and the deadline is consumed before the last stage starts.
- Silent stale publish: partial publish without a freshness stamp, so users act on old numbers.
- SLO on a stage nobody consumes while the end product has none.
Trade-offs
| Lever | Gains | Pays |
|---|---|---|
| Fewer stages | Higher chain reliability, simpler lineage | Less clean layering |
| More slack | Absorbs slips | Earlier starts, bigger clusters |
| Partial publish | Always something on time | Consumers must read staleness |
When not to use it
For reports nobody needs by a deadline, the arithmetic is wasted effort. Use it where lateness has a price: finance close, partner feeds, model retrains.
Interview question
Q: A product SLA of 99% is required from a six-stage pipeline. What do you do?
What a strong answer covers: the product of probabilities and the required per-stage rate (~99.83%); correlation and shared fate; reducing stages; slack and partial publish; defining the SLO on the product; and the error-budget conversation with the business.
Quick check
Quiz: Four stages at 99.5% each: chain on-time rate? About 98.0% (0.995^4).
Flashcard: Why do added layers lower freshness reliability? — Each serial stage multiplies in another probability below one.