Audit Log Service  ·  View 30 of 40  ·  6 · Operations

Two Regions, and Custody That Belongs to Neither

Active-passive failover, one-way replication, and the reason the checkpoint ledger sits outside both regions.

Editable source SVG draw.io All views
West Europe — active Ingest Plane 25 k/s steady Custody Plane sealing, chaining Query Plane full index Segment Store locked, ZRS North Europe — warm standby Ingest Plane 20% warm · RTO 15 min Custody Plane idle, resumes chain Query Plane lagging · RTO 4 h Segment Replica locked at destination Continuous replication — one way, no delete right at the destination Object Replication segments · RPO ≈ 0 MirrorMaker 2 buffer · 72 h ClickHouse Replica rebuildable anyway PostgreSQL Replica registry, holds Custody independent of both regions Confidential Ledger own subscription Timestamp Authority outside the estate Published Public Key verifier needs no API Public Witness Log Rekor · phase 3 locked on arrival signed heads Two Regions, and a Custody Chain That Belongs to Neither Application we own Data store Queue / topic External / third party Security / platform event / async synchronous A region failover changes which region seals; it changes nothing about what has already been sealed. The chain continues in the second region from the last checkpoint the ledger holds, and the ledger is in neither region's subscription — which is what stops a regional administrator from being able to rewrite both sides of the evidence. v 1.0 · owner Data & AI Global Practice · date 2026-09

Decisions

  • Active-passive, not active-active. Two regions sealing independently would need a cross-region ordering claim the requirement never asks for, and would double the surface on which a chain can fork.
  • Replication is one way and the destination grants no delete right. Segments arrive already under a locked policy at the destination, so a replica is not a softer copy of a hard original.
  • A failover changes which region seals; it changes nothing about what has been sealed. The chain resumes in the second region from the last head the ledger holds — and the ledger is in neither region's subscription.

Recovery targets

  • RTO 15 minutes for ingest, 4 hours for query. The asymmetry is deliberate: accepting events is the promise, finding them quickly is a service level.
  • RPO 0 for acknowledged events. Segment durability 11 nines annual with continuous cross-region replication.
  • Buffer state replicates via MirrorMaker 2 within the 72 h window; the index replica is convenience, since the index is rebuildable anyway.

Risks

  • MirrorMaker introduces its own lag, so a hard region loss can strand events acknowledged in the final seconds before failure. Meeting RPO 0 strictly requires the producer to hold its spool until a seal receipt is available — which is exactly why the two-stage receipt exists, and why fail-closed classes should poll for it.
◀  Deployment — One Region, Three Zones All views Shipping the Service, Changing the Rules  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 40 views · Microsoft Azure and open source30-multi-region-and-anchoring