Event-Driven Notification Platform  ·  View 18 of 26  ·  5 · Operations

Deployment and Failure Domains

What runs where, what survives a zone loss, and what a region loss actually costs.

Editable source SVG draw.io All views
Primary region · eu-west-1 · active
Primary region · eu-west-1 · active
Zone A
Zone A
Kubernetes nodes
8 × 16 vCPU
Kubernetes nodes...
Kafka brokers
4 × NVMe
Kafka brokers...
PostgreSQL primary
CloudNativePG
PostgreSQL primary...
ClickHouse shard 1
ClickHouse shard 1
Zone B
Zone B
Kubernetes nodes
8 × 16 vCPU
Kubernetes nodes...
Kafka brokers
4 × NVMe
Kafka brokers...
PostgreSQL sync replica
quorum commit
PostgreSQL sync replica...
ClickHouse shard 2
ClickHouse shard 2
Zone C
Zone C
Kubernetes nodes
6 × 16 vCPU
Kubernetes nodes...
Kafka brokers
4 × NVMe
Kafka brokers...
PostgreSQL async replica
read scale
PostgreSQL async replica...
ClickHouse shard 3
ClickHouse shard 3
DR region · eu-central-1 · warm standby
DR region · eu-central-1 · warm standby
Standby footprint
Standby footprint
Kubernetes
30% · scales on promote
Kubernetes...
Kafka
MirrorMaker 2 target
Kafka...
PostgreSQL
streaming replica · lag under 5 s
PostgreSQL...
MinIO
cross-region replication
MinIO...
Vault replica
performance standby
Vault replica...
Global load balancer
health-based failover
Global load balancer...
Harbor registry
geo-replicated
Harbor registry...
Channel providers
region-agnostic endpoints
Channel providers...
primary traffic
primary traffic
on promotion · RTO 30 min
on promotion · RTO 30 min
MirrorMaker 2
MirrorMaker 2
streaming replication
streaming replication
synchronous commit
synchronous commit
Deployment and Failure Domains
Deployment and Failure Domains
Security / platform
Security / platform
Queue / topic
Queue / topic
Data store
Data store
Interface / broker
Interface / broker
External / third party
External / third party
synchronous
synchronous
failure / alternate
failure / alternate
event / async
event / async
Kafka with replication factor 3 and min.insync.replicas 2 gives RPO 0 in-region for an acknowledged event. Losing one zone costs capacity, not data; losing the region costs at most five minutes of unreplicated events.
Kafka with replication factor 3 and min.insync.replicas 2 gives RPO 0 in-region for an acknowledged event. Losing one zone costs capacity, not data; losing the region costs at most five minutes of unreplicated events.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display

Decisions

  • Three availability zones with a Kafka quorum spanning them, so a zone loss costs capacity and not data
  • Warm standby rather than active-active: active-active would need conflict resolution on notification state for a 30-minute recovery target that a warm standby already meets
  • The DR region runs at 30 percent and scales on promotion, which trades recovery speed for about 70 percent of a second region's cost

Recovery objectives

  • RPO 0 in-region for an acknowledged event: replication factor 3, min.insync.replicas 2, acks=all
  • RPO 5 minutes cross-region, bounded by MirrorMaker lag
  • RTO 30 minutes, dominated by DNS propagation and standby scale-out, not by data movement
  • PostgreSQL point-in-time recovery to 7 days, tested monthly

Risks

  • Provider credentials must be present in the DR region before an incident; Vault performance replication covers this and is the single most likely thing to be missed in a real failover
  • MirrorMaker offset translation means replayed consumer positions after failover are approximate; the recovery runbook replays from the archive by time range instead