Audit Log Service  ·  View 14 of 40  ·  4 · Data

Data Architecture

Four tiers with four different durability contracts, ordered by what losing each one actually costs.

Editable source SVG draw.io All views
System of record — losing any of this is unrecoverable Immutable Segments Blob · 11 nines · RPO 0 Checkpoint Ledger separate subscription Disposal Certificates records inside the log Durable hand-off — loss of an acknowledged event here is a Sev-1 defect Kafka Buffer 72 h · RF 3 · isr 2 Quarantine Topic sealed like any other Derived and rebuildable — deleting all of it loses no record ClickHouse Index 90 d default, 13 mo max Segment Registry PostgreSQL · rebuildable Absence Filters for non-existence proofs Ephemeral — bounded loss accepted, stated on the contract Dedup Window Valkey · 24 h TTL Quota Counters reset on loss Result Cursors 15 min resumable seal ≤ 60 s rebuild Data Architecture — Four Tiers, Four Durability Contracts Data store Queue / topic event / async batch Losing the dedup window costs duplicate records, which are visible and reconcilable. Losing the index costs query latency for the length of a rebuild — 2.6 TB/day re-read from segments. Losing a segment costs the audit its meaning, which is why only that tier gets a locked retention policy. v 1.0 · owner Data & AI Global Practice · date 2026-09

Decisions

  • Only one tier is the system of record. Everything else is a convenience with a stated recovery path, and the whole architecture is arranged so that saying this out loud is true rather than aspirational.
  • The index is authoritative for nothing. Anything that would be lost by deleting it entirely is a design defect.
  • The checkpoint ledger is in the system-of-record tier despite being tiny, because losing it costs the ability to prove anything about the segments that survive.

Recovery contracts

  • Segments: RPO 0, 11 nines annual durability, continuous cross-region replication, locked retention policy.
  • Buffer: loss of an acknowledged event is a Sev-1 defect, not a degradation. 72 h replay window.
  • Index and registry: fully rebuildable from segments — 2.6 TB/day re-read, roughly 14 h for a 90-day window.
  • Ephemeral tier: bounded loss accepted and stated. Duplicate records and reset quota counters are the visible consequence.

Assumptions

  • Absence filters (for proofs of non-existence, phase 3) are treated as derived and rebuildable. If a regulator ever requires a non-existence proof to be independently attested, they move up a tier and the cost model changes.
◀  The Shape of the Hash Chain All views The Audit Record  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 40 views · Microsoft Azure and open source14-data-architecture