Real-Time Analytics Platform · View 19 of 21 · Operations

Reliability and Recovery Loop

The loop the platform runs when something goes wrong, and the point at which retained raw events turn a detection into a recovery.

Editable source SVG draw.io All views

Why this closes

Detection without replay leaves a permanent gap; replay is what makes the loop close
Reconciliation is the exit condition — an incident is not resolved until variance is verified
Every stage has a named runbook and an owning rota

Failure modes covered

Processing defect: contain, fix, replay from bronze into a shadow table, swap
Sink unavailability: checkpoint stalls, Event Hubs absorbs, recovery resumes from offset
Slow partition: watermark skew alert fires before the freshness SLO is breached

What is not covered

Producer-side data quality defects — the platform records them, it cannot correct them
Loss of bronze itself, which is mitigated by geo-redundant storage and immutability
Regional failover, which is a separate declared procedure rather than this loop

◀ Observability and Service Level Objectives All views Security Architecture — Trust Zones ▶