Audit Log Service  ·  View 28 of 40  ·  5 · Runtime

Losing the Index

Six stages proving the claim the whole set rests on — that deleting the index loses no record.

Editable source SVG draw.io All views
Detect Reconciler Divergence count or root mismatch Node or Disk Loss shard unavailable Decide Partition Repair minutes Full Rebuild 90 d ≈ 14 h Tenant Priority largest and regulated first Degrade Trino Over Segments slower, still correct Query SLO Suspended stated, not hidden Ingest Untouched still 99.99% Rebuild Read Segments 650 GB/day compressed Parallel Insert 24 indexer pods Absence Filters rebuilt with it Reconcile Count per Segment index vs registry Root per Segment index vs chain Restore Completeness Bound advances to now SLO Resumed p95 2 s again mismatch, re-read Losing the Index — and Why It Loses No Record Application we own Risk / gap Decision point Opportunity Data store failure / alternate Question 4 answered by making divergence measurable rather than impossible. Two stores are kept, and every query result states the sealed segment it is complete to — so the index is never quietly treated as authoritative, and a record that exists but cannot yet be found is visible as such. v 1.0 · owner Data & AI Global Practice · date 2026-09

Question 4 answered

  • Two stores, with divergence made measurable rather than impossible. A single ledger-style store would avoid reconciliation entirely and would not hold a 2-second point query or a 30-second thirteen-month aggregate.
  • Divergence is detected by comparing per-segment record counts and Merkle roots against what the index holds. A record in a segment but missing from the index is repaired automatically; a record in the index but not in a segment is an incident.
  • Every query result states the sealed segment it is complete to, so the index is never quietly treated as authoritative. To an auditor a record that exists but cannot be found is missing — the completeness bound is what turns that into a stated, bounded lag.

Rebuild economics

  • A full 90-day rebuild reads roughly 58 TB compressed and takes about 14 h at 24 indexer pods; a 13-month rebuild is about four times that.
  • Rebuild order is by tenant priority, largest and most regulated first, so the tenants who will ask are served first.
  • Throughout, ingest is untouched and still meeting 99.99%. The query SLO is explicitly suspended and users are told, rather than quietly missed.

Risks

  • A 13-month index rebuild takes over two days, during which investigative queries run on the archive path. If that is unacceptable to the business, the answer is a warm standby index in the second region — priced but not in the MVP.
◀  When the Audit Plane Is Unavailable All views Deployment — One Region, Three Zones  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 40 views · Microsoft Azure and open source28-index-loss-and-rebuild