Event-Driven Notification Platform  ·  View 11 of 26  ·  3 · Data

Data Architecture and Retention

What is authoritative, what is rebuildable, and how long each thing lives.

Editable source SVG draw.io All views
Authoritative — cannot be rebuilt, must be backed up
Authoritative — cannot be rebuilt, must be backed up
Configuration
Configuration
Config Store
rules · tenants · providers
Config Store...
Template versions
every version kept
Template versions...
Consent and identity
Consent and identity
Preference Store
Postgres · RLS
Preference Store...
PII Vault
pgcrypto · per-tenant DEK
PII Vault...
Audit Log
MinIO object lock · 7 y
Audit Log...
Operational — rebuildable from the event log
Operational — rebuildable from the event log
Hot
Hot
Notification Store
partitioned · 30 d
Notification Store...
Redis
dedup 7 d · counters
Redis...
Kafka topics
3 to 7 d retention
Kafka topics...
Analytical
Analytical
Delivery Store
ClickHouse · 90 d raw
Delivery Store...
Rolled-up aggregates
13 months
Rolled-up aggregates...
Archive — immutable, cheap, queryable
Archive — immutable, cheap, queryable
Long term
Long term
Event Archive
Iceberg · 400 d
Event Archive...
Notification archive
Iceberg · 13 months
Notification archive...
Backup target
cross-region · PITR 7 d
Backup target...
Erasure job
purge cascade within 30 d
Erasure job...
Trino
archive queries · read only
Trino...
5 min micro-batch
5 min micro-batch
nightly at 30 d
nightly at 30 d
hourly
hourly
hard delete
hard delete
continuous WAL
continuous WAL
SQL read
SQL read
Data Architecture and Storage Zones
Data Architecture and Storage Zones
Data store
Data store
Security / platform
Security / platform
Queue / topic
Queue / topic
Application we own
Application we own
batch
batch
failure / alternate
failure / alternate
event / async
event / async
synchronous
synchronous
The retention numbers are the design. Anything in the operational zone can be lost and reconstructed from the archive; nothing in the authoritative zone can, which is what backup and DR are sized around.
The retention numbers are the design. Anything in the operational zone can be lost and reconstructed from the archive; nothing in the authoritative zone can, which is what backup and DR are sized around.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display

The dividing line

  • Authoritative data — configuration, consent, templates, audit — cannot be reconstructed and defines the backup scope
  • Operational data can be rebuilt from the event archive, so it is sized for speed rather than for durability
  • This is why backup is small and cheap while the platform still stores about a billion delivery rows a day

Retention

  • Kafka 3 to 7 days by tier; DLQ 30 days
  • Notification records hot 30 days in PostgreSQL, then Iceberg for 13 months
  • Delivery attempts 90 days raw in ClickHouse, rolled up to 13 months
  • Raw events 400 days in Iceberg; audit 7 years under object lock; PII deletable on request within 30 days

Assumptions

  • 400 days for raw events covers a full year plus a reconciliation window; confirm against the tenant contract
  • Object lock on the audit bucket satisfies the write-once requirement without a separate WORM appliance