Usage Metering & Billing Pipeline  ·  View 22 of 22  ·  Assurance

Failure Classes

Eight assumed failures, how each is detected and handled, and what is knowingly left.

Editable source SVG draw.io All views
Detected by Handled by Residual risk Duplicate delivery Dedup hit rate Idempotency on event id Retry past 7 days Late arrival Watermark lag 72 h lateness bound Beyond bound: restate Lost before acknowledgement Producer gap report Durable local buffer Crash with full buffer Malformed or drifted event Quarantine rate Quarantine then replay Drift nobody reads Pricing misconfiguration Blast-radius preview Two-person gate Both approvers wrong Hot account Partition skew Sub-partition and quota Skew past sub-partition Ledger unavailable Accept error rate Reject; producer retains Outage past buffer depth Downstream unavailable Outbox depth Durable outbox, replay Invoice late to payment Failure Classes — Detection, Handling, Residual Risk Every residual risk here is accepted deliberately and is either bounded by a stated number or escalates into a restatement, which is the one failure path the architecture is built to make routine. v 1.0 · owner Architecture Practice · date 2026-10

The principle

  • Every residual risk here is accepted deliberately and either bounded by a stated number or escalated into a restatement — the one failure path the architecture is built to make routine rather than exceptional.
  • Ingestion fails closed and serving fails open: where durability cannot be guaranteed the event is rejected so the producer keeps it, and the product keeps serving regardless of the metering path's health.
  • Nothing here is handled by retry alone. Retry without idempotency is how a billing pipeline double-charges.

Bounded by a number

  • Duplicate past 7 days, late past 72 hours, outage past producer buffer depth — each residual risk is the far side of a stated assumption, and each is arguable because the number is written down.
  • Accuracy target of 1 in 10^6 events is what makes these residuals measurable rather than rhetorical.

The unbounded ones

  • Two residual risks have no number: both approvers wrong on a price change, and schema drift nobody reads. The first is a human control and the second is a monitoring discipline, and neither is engineered away by this design.
  • Skew beyond a sub-partition is the scale limit of the partitioning scheme and will arrive with the largest customer, not gradually.