Audit Log Service  ·  View 40 of 40  ·  7 · Assurance

Named Failure Modes

Sixteen failures in four categories, sorted by what the service is permitted to do about each.

Editable source SVG draw.io All views
Absorbed — the producer sees nothing Ingest Timeout retry + idempotency key Duplicate Delivery one record, 24 h window Zone Loss isr ≥ 2, transparent Broker Restart producer backoff Degraded — in the declared order, never the reverse Key Service Down ingest on, sealing pauses Index Loss Trino path, SLO suspended Region Loss RTO 15 min / 4 h Sealer Backlog buffer absorbs 72 h Contained and made visible — the gap is itself a record Producer Outage spool 1 h, gap marker Poison Record quarantine, no head-of-line Clock Skew flagged beyond 5 min Seal Gap re-sealed, never discarded Incident — never handled automatically Insider Deletion Attempt prevented, alerted Segment Divergence Sev-1, no auto-repair Acknowledged, Never Sealed Sev-1 defect, RPO 0 Checkpoint Signature Fails key or ledger compromise unless roots differ Named Failure Modes and Their Required Handling Application we own Risk / gap failure / alternate Index divergence is repaired from segments and is routine; segment divergence is repaired by nobody and is an incident. The distinction is the whole architecture in one line: the derived thing may be rebuilt, and the thing it was derived from may not be touched. v 1.0 · owner Data & AI Global Practice · date 2026-09

The categories are the decision

  • Absorbed: the producer sees nothing. Retries, duplicates and a lost zone all fall here, which is what the durability contract was bought for.
  • Degraded: in the declared order and never the reverse. A key service outage pauses sealing while ingest continues; an index loss suspends the query SLO while records keep landing.
  • Contained and visible: a spool overflow, a poison record, a skewed clock or a seal gap all produce a record about themselves. The gap is never a silence.
  • Incident: an attempted deletion, a segment divergence, an acknowledged record that never sealed, a checkpoint signature that will not verify. None of these is ever handled automatically.

The line that matters

  • Index divergence is repaired from segments and is routine. Segment divergence is repaired by nobody and is a Sev-1.
  • That one distinction is the whole architecture: the derived thing may be rebuilt, and the thing it was derived from may not be touched — by anyone, including the people repairing it.

Open questions for the sponsor

  • Confirm the equivalence of Blob locked immutability to the erasure standard the applicable regulators accept, and whether crypto-shredding satisfies them or physical deletion is required.
  • Agree the fail-closed event class list with the owners of the 410 producing services, because it changes their SLOs and not only ours.
  • Renegotiate the $0.09 per million target, or agree explicitly which of DR, HSM-backed signing and the separately administered ledger is being given up to approach it.
◀  The Operators Are In Scope All views
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 40 views · Microsoft Azure and open source40-failure-modes