Event-Driven Notification Platform

Architecture Views

26 views, in reading order. Every view ships three ways: an HTML page, an SVG that re-opens in diagrams.net fully editable, and draw.io source.

Read the set in order. Acts 1 and 2 fix the boundary and the structure; act 3 says what is stored, who owns it and what may be sent to whom; act 4 shows what actually happens when an event arrives, including when it fails; act 5 covers running it; act 6 covers why it is safe and what breaks. Six decisions carry the design: events and notifications are separate domains with separate storage; rules are evaluated in a stateful stream job rather than per-event RPC; durable workflow state is used only for the 15 percent of traffic that needs delay, fallback or multi-step behaviour; three priority tiers run end to end so a marketing burst cannot delay a one-time passcode; personally identifiable data is resolved at render time and never touches the bus or the archive; and at-least-once with idempotent consumers is the stated contract, with deduplication that fails open so the platform duplicates rather than drops.

1 · Context and scope

The boundary: who publishes, who receives, and what the platform never owns.
01
People
People
Tenant Administrator
Tenant Administrator
Support Engineer
Support Engineer
Recipient
Recipient
Event producers
Event producers
Order Service
OrderCreated · OrderShipped
Order Service...
Payment Service
PaymentFailed
Payment Service...
Identity Service
OTP · security alert
Identity Service...
Campaign Manager
bulk audiences
Campaign Manager...
Delivery providers
Delivery providers
Email providers
Postal · Amazon SES
Email providers...
SMS aggregators
Jasmin · Twilio
SMS aggregators...
FCM and APNs
FCM and APNs
Slack and Teams
Slack and Teams
Partner webhooks
Partner webhooks
Notification Platform
events in · notifications out
Notification Platform...
Enterprise services
Enterprise services
Keycloak
OIDC · workload identity
Keycloak...
Customer Master
system of record
Customer Master...
Grafana LGTM
metrics · logs · traces
Grafana LGTM...
publishes events
publishes events
publishes events
publishes events
P0 events
P0 events
batch submit
batch submit
rules · templates
rules · templates
traces a notification
traces a notification
receives · opts out
receives · opts out
SMTP · REST
SMTP · REST
SMPP 3.4
SMPP 3.4
HTTP/2 push
HTTP/2 push
chat webhook
chat webhook
HMAC POST
HMAC POST
authenticates producers
authenticates producers
contact lookup
contact lookup
telemetry
telemetry
Event-Driven Notification Platform — System Context
Event-Driven Notification Platform — System Context
Person or role
Person or role
External / third party
External / third party
Security / platform
Security / platform
event / async
event / async
batch
batch
synchronous
synchronous
Out of scope: producing the business events, owning customer contact data, and anything past the provider accept response.
Out of scope: producing the business events, owning customer contact data, and anything past the provider accept response.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
System Context Who publishes events, who receives notifications, and where the platform boundary sits. HTML page SVG draw.io
02
Produce
Produce
Business applications
no PII in payload
Business applications...
Campaign Manager
bulk audiences
Campaign Manager...
Ingest
Ingest
Envoy Gateway
mTLS · JWT · quota
Envoy Gateway...
Event Ingest API
Go · ack p95 80 ms
Event Ingest API...
Schema Registry
Avro · backward only
Schema Registry...
Distribute
Distribute
Apache Kafka
3 priority tiers · RF 3
Apache Kafka...
Event Archive
Iceberg · 400 d
Event Archive...
Decide
Decide
Rules Engine
Flink · CEL conditions
Rules Engine...
Preference Service
consent · quiet hours
Preference Service...
Orchestrator
Temporal workflows
Orchestrator...
Compose
Compose
Template Service
versioned · localised
Template Service...
Render Service
Handlebars · MJML
Render Service...
Rate Limiter
Redis token bucket
Rate Limiter...
Deliver
Deliver
Channel Workers
6 channels · Go
Channel Workers...
Providers
2 per channel
Providers...
Delivery Store
ClickHouse · 90 d
Delivery Store...
5 min micro-batch
5 min micro-batch
receipts
receipts
High-Level Architecture
High-Level Architecture
External / third party
External / third party
Interface / broker
Interface / broker
Application we own
Application we own
Security / platform
Security / platform
Queue / topic
Queue / topic
Data store
Data store
batch
batch
event / async
event / async
Retry, channel fallback, scheduling and every failure path are deliberately omitted here — acts 4 and 6 carry them.
Retry, channel fallback, scheduling and every failure path are deliberately omitted here — acts 4 and 6 carry them.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
High-Level Architecture The shape of the platform in one picture, for the reader who will see only one. HTML page SVG draw.io

2 · Structure

The layers, the deployable units, every interface, the channel contract, how content is governed, and what tenants share.
03
Experience
Experience
Admin Console
React
Admin Console...
Recipient Inbox
Centrifugo WebSocket
Recipient Inbox...
Producer SDKs
Go · Java · Python
Producer SDKs...
API
API
Event API
POST /v1/events
Event API...
Notification API
direct send
Notification API...
Preference API
Preference API
Template API
Template API
Admin API
replay · config
Admin API...
Decision
Decision
Rules Engine
CEL over event + context
Rules Engine...
Orchestrator
multi-step · fallback
Orchestrator...
Preference Service
Preference Service
Scheduler
timers · cron · digests
Scheduler...
Composition
Composition
Template Service
draft · published · rollback
Template Service...
Render Service
Render Service
Recipient Profile
subject_ref to contact
Recipient Profile...
Localisation
locale · timezone
Localisation...
Delivery
Delivery
Channel Router
Channel Router
Channel Adapter SPI
one contract per channel
Channel Adapter SPI...
Delivery Workers
6 channel pools
Delivery Workers...
Provider Clients
2 per channel
Provider Clients...
Messaging
Messaging
Apache Kafka
Strimzi · KRaft
Apache Kafka...
Schema Registry
Apicurio · Avro
Schema Registry...
DLQ Topics
per tier · 30 d
DLQ Topics...
Data
Data
PostgreSQL
notifications · config
PostgreSQL...
ClickHouse
delivery attempts
ClickHouse...
Redis
dedup · counters
Redis...
MinIO and Iceberg
archive · audit
MinIO and Iceberg...
Platform
Platform
Kubernetes
Kubernetes
Istio ambient
mTLS everywhere
Istio ambient...
HashiCorp Vault
tenant credentials
HashiCorp Vault...
Keycloak
OIDC
Keycloak...
Grafana LGTM
observability
Grafana LGTM...
REST + OIDC
REST + OIDC
produce acks=all
produce acks=all
notification intent
notification intent
render request
render request
composed message
composed message
writes
writes
cached preferences
cached preferences
upward call
upward call
Layered Architecture
Layered Architecture
Application we own
Application we own
Interface / broker
Interface / broker
Queue / topic
Queue / topic
Security / platform
Security / platform
Data store
Data store
synchronous
synchronous
event / async
event / async
failure / alternate
failure / alternate
Dependencies point downward with one exception, drawn in red: the in-app worker pushes into the experience layer. Accepted because the alternative is client polling at 750 M notifications a day.
Dependencies point downward with one exception, drawn in red: the in-app worker pushes into the experience layer. Accepted because the alternative is client polling at 750 M notifications a day.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Layered Architecture What depends on what, and the single dependency that points the wrong way. HTML page SVG draw.io
04
Notification Platform · Kubernetes with Istio ambient mesh
Notification Platform · Kubernetes with Istio ambient mesh
Edge and API
Edge and API
Envoy Gateway
mTLS · JWT · quota
Envoy Gateway...
Event Ingest API
Go · 12 pods
Event Ingest API...
Notification API
Go · 6 pods
Notification API...
Admin API
Go · 3 pods
Admin API...
Admin Console
React · static
Admin Console...
Decision plane
Decision plane
Rules Engine
Flink · 24 task managers
Rules Engine...
Dedup Processor
Flink keyed state · 7 d
Dedup Processor...
Orchestrator
Temporal · 40 workers
Orchestrator...
Preference Service
Go · 8 pods
Preference Service...
Scheduler
Temporal cron
Scheduler...
Composition plane
Composition plane
Template Service
Go · 4 pods
Template Service...
Render Service
Go · 20 pods
Render Service...
Recipient Profile
Go · 8 pods
Recipient Profile...
Rate Limiter
Redis Lua
Rate Limiter...
Delivery plane
Delivery plane
Email Worker
Go · 30 pods
Email Worker...
SMS Worker
Go · 12 pods
SMS Worker...
Push Worker
Go · 16 pods
Push Worker...
In-App Worker
Go · 8 pods
In-App Worker...
Webhook Worker
Go · 10 pods
Webhook Worker...
Chat Worker
Go · 4 pods
Chat Worker...
Stateful services
Stateful services
Apache Kafka
Strimzi · 12 brokers
Apache Kafka...
PostgreSQL
CloudNativePG · HA
PostgreSQL...
ClickHouse
3 shards × 2 replicas
ClickHouse...
Redis Cluster
6 nodes
Redis Cluster...
MinIO
Iceberg warehouse
MinIO...
Keycloak
OIDC
Keycloak...
HashiCorp Vault
tenant credentials
HashiCorp Vault...
Grafana LGTM
observability
Grafana LGTM...
Channel providers
email · SMS · push · chat
Channel providers...
Customer Master
contact lookup
Customer Master...
HTTP/2 · mTLS
HTTP/2 · mTLS
produce acks=all
produce acks=all
consume evt.valid
consume evt.valid
notif.intent
notif.intent
gRPC render
gRPC render
resolve contact
resolve contact
deliver topic
deliver topic
SMTP · REST
SMTP · REST
lease credential
lease credential
gRPC · cached 24 h
gRPC · cached 24 h
Container and Component Architecture
Container and Component Architecture
Interface / broker
Interface / broker
Application we own
Application we own
Queue / topic
Queue / topic
Data store
Data store
Security / platform
Security / platform
External / third party
External / third party
synchronous
synchronous
event / async
event / async
Pod counts are the steady-state floor at 5 000 events/s. Every plane scales independently; view 20 gives the signals and the ceilings.
Pod counts are the steady-state floor at 5 000 events/s. Every plane scales independently; view 20 gives the signals and the ceilings.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Container and Component Architecture Every deployable unit, its technology, its replica floor, and the protocol on every hop. HTML page SVG draw.io
05
Inbound — who calls us
Inbound — who calls us
Business applications
Business applications
Campaign Manager
Campaign Manager
Native Kafka producers
Native Kafka producers
Transactional callers
Transactional callers
Admin Console
Admin Console
Recipient apps
Recipient apps
Template GitOps
Template GitOps
Platform
Platform
Notification Platform
OpenAPI 3.1 · AsyncAPI 3
Notification Platform...
Outbound — what we call
Outbound — what we call
Postal and Amazon SES
Postal and Amazon SES
Jasmin and Twilio
Jasmin and Twilio
FCM and APNs
FCM and APNs
Slack and Teams
Slack and Teams
Partner webhooks
Partner webhooks
Customer Master
Customer Master
Keycloak
Keycloak
HashiCorp Vault
HashiCorp Vault
POST /v1/events
POST /v1/events
bulk API · 10 k/batch
bulk API · 10 k/batch
SASL/SCRAM · Avro
SASL/SCRAM · Avro
POST /v1/notifications
POST /v1/notifications
REST + OIDC
REST + OIDC
WebSocket inbox
WebSocket inbox
signed bundle
signed bundle
SMTP · REST
SMTP · REST
SMPP 3.4
SMPP 3.4
HTTP/2
HTTP/2
chat webhook
chat webhook
HMAC POST
HMAC POST
gRPC lookup
gRPC lookup
JWKS
JWKS
1 h lease
1 h lease
Integration and Interface Catalogue
Integration and Interface Catalogue
External / third party
External / third party
Application we own
Application we own
Interface / broker
Interface / broker
Security / platform
Security / platform
synchronous
synchronous
batch
batch
Every inbound interface is versioned and contract-tested in CI. Every outbound provider has a documented second vendor; none is a platform dependency.
Every inbound interface is versioned and contract-tested in CI. Every outbound provider has a documented second vendor; none is a platform dependency.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Integration and Interface Catalogue Every way something enters or leaves the platform, with protocol, cadence and owner. HTML page SVG draw.io
06
Resolve address
Resolve address
Render
Render
Throttle
Throttle
Dispatch
Dispatch
Confirm
Confirm
Email
Email
Verified address
from PII Vault
Verified address...
MJML to HTML
plus text alternative
MJML to HTML...
200 msg/s per tenant
200 msg/s per tenant
Postal MTA
SES on failover
Postal MTA...
Bounce and open
provider webhook
Bounce and open...
SMS
SMS
E.164 MSISDN
country validated
E.164 MSISDN...
GSM-7 segmenter
160 char parts
GSM-7 segmenter...
Carrier TPS cap
per route
Carrier TPS cap...
Jasmin SMSC
SMPP bind
Jasmin SMSC...
DLR receipt
DLR receipt
Push
Push
Device token set
per platform
Device token set...
FCM and APNs payload
4 KB limit
FCM and APNs payload...
1 per device per min
1 per device per min
FCM HTTP v1
APNs direct
FCM HTTP v1...
Token invalidation
prunes dead devices
Token invalidation...
In-app
In-app
Inbox identity
no external contact
Inbox identity...
JSON card
action buttons
JSON card...
None
pull-limited
None...
Centrifugo
WebSocket fan-out
Centrifugo...
Read receipt
Read receipt
Webhook
Webhook
Endpoint and secret
Vault path
Endpoint and secret...
JSON envelope
stable schema
JSON envelope...
50 rps per endpoint
50 rps per endpoint
HMAC-signed POST
customer endpoint
HMAC-signed POST...
2xx or retry
5 attempts
2xx or retry...
Chat
Chat
Channel or user ID
Channel or user ID
Block Kit blocks
Adaptive Card for Teams
Block Kit blocks...
Workspace tier cap
Workspace tier cap
Slack chat.postMessage
Teams Graph
Slack chat.postMessage...
Message timestamp
edit and thread
Message timestamp...
Channel Abstraction — One Contract, Six Adapters
Channel Abstraction — One Contract, Six Adapters
Application we own
Application we own
External / third party
External / third party
Queue / topic
Queue / topic
Adding a channel means implementing one interface — resolve, render, throttle, dispatch, confirm — plus a provider client. No core service changes.
Adding a channel means implementing one interface — resolve, render, throttle, dispatch, confirm — plus a provider client. No core service changes.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Channel Abstraction The five-step contract every channel implements, and what differs between them. HTML page SVG draw.io
07
Author
Author
Tenant Administrator
Tenant Administrator
Draft version
semver · never sent
Draft version...
Variable contract
declared and typed
Variable contract...
Validate
Validate
Syntax and variables
no undeclared token
Syntax and variables...
Output sanitisation
no raw HTML injection
Output sanitisation...
Channel constraints
SMS length · push 4 KB
Channel constraints...
Preview
Preview
Render preview
synthetic data
Render preview...
Test send
allow-listed recipients
Test send...
Locale matrix
fallback to en-GB
Locale matrix...
Approve
Approve
Second approver
required for marketing
Second approver...
Consent copy check
unsubscribe present
Consent copy check...
Publish
Publish
Published version
immutable · addressable
Published version...
Template Store
all versions kept
Template Store...
Render cache warm
60 s propagation
Render cache warm...
Operate
Operate
Notification binds version
pinned at decision time
Notification binds version...
Rollback
repoint alias · no redeploy
Rollback...
Audit record
who · what · when
Audit record...
rejected
rejected
changes requested
changes requested
previous version
previous version
Template and Content Lifecycle
Template and Content Lifecycle
Person or role
Person or role
Application we own
Application we own
Decision point
Decision point
Data store
Data store
Security / platform
Security / platform
failure / alternate
failure / alternate
event / async
event / async
A notification records the exact template version it rendered, so a rollback never rewrites what was already sent and a support query can reproduce the original message.
A notification records the exact template version it rendered, so a rollback never rewrites what was already sent and a support query can reproduce the original message.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Template and Content Lifecycle How a message goes from draft to published, and how a bad one is taken back. HTML page SVG draw.io
08
Shared compute — one platform, many tenants
Shared compute — one platform, many tenants
Shared
Shared
Kafka cluster
tiered topics
Kafka cluster...
Kubernetes node pools
Kubernetes node pools
Flink job
keyed by tenant_id
Flink job...
Channel worker pools
shared tier
Channel worker pools...
Per tenant
Per tenant
Rule set
tenant-scoped CEL
Rule set...
Template set
Template set
Provider credentials
Vault path per tenant
Provider credentials...
Quota
events/s and notif/day
Quota...
Data isolation
Data isolation
Logical — every tenant
Logical — every tenant
Notification Store
RLS on tenant_id
Notification Store...
Preference Store
RLS on tenant_id
Preference Store...
Delivery Store
tenant_id sort key
Delivery Store...
PII Vault
per-tenant data key
PII Vault...
Physical — dedicated tier
Physical — dedicated tier
Dedicated topics
top 5 tenants
Dedicated topics...
Dedicated worker pool
taint and toleration
Dedicated worker pool...
Dedicated database
regulated tenants
Dedicated database...
Noisy-neighbour controls
Noisy-neighbour controls
Admission
Admission
Gateway quota
per-tenant rps + burst
Gateway quota...
Priority tiering
P0 · P1 · P2
Priority tiering...
Payload cap
64 KB hard
Payload cap...
Execution
Execution
Consumer concurrency cap
per tenant per channel
Consumer concurrency cap...
Provider TPS budget
fair share
Provider TPS budget...
Circuit breaker
tenant × provider
Circuit breaker...
Shared Flink job
one hot key lags a partition
Shared Flink job...
Per-tenant metrics
lag · spend · error rate
Per-tenant metrics...
classified
classified
tier topic
tier topic
skew risk
skew risk
trip events
trip events
Multi-Tenancy — What Is Shared and What Is Not
Multi-Tenancy — What Is Shared and What Is Not
Queue / topic
Queue / topic
Security / platform
Security / platform
Application we own
Application we own
Data store
Data store
Interface / broker
Interface / broker
Decision point
Decision point
Risk / gap
Risk / gap
synchronous
synchronous
event / async
event / async
failure / alternate
failure / alternate
Tenants share compute by default and buy isolation where they need it. The one unsolved risk is drawn in red: a single hot tenant key can lag a Flink partition, mitigated by key salting on the P2 tier only.
Tenants share compute by default and buy isolation where they need it. The one unsolved risk is drawn in red: a single hot tenant key can lag a Flink partition, mitigated by key salting on the P2 tier only.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Multi-Tenancy and Noisy-Neighbour Control What every tenant shares, what each one gets alone, and what stops one from hurting the rest. HTML page SVG draw.io

3 · Data

Three separate domains — event, notification, delivery — where each lives, for how long, and what consent permits.
09
Origin
Origin
Producer application
owns the business fact
Producer application...
Event envelope
subject_ref, not PII
Event envelope...
Ingest
Ingest
Validate and stamp
event_id · received_at
Validate and stamp...
Schema check
Avro backward
Schema check...
Invalid events
evt.invalid · 30 d
Invalid events...
Backbone
Backbone
Event topics
key tenant:entity · 7 d
Event topics...
Event Archive
Iceberg · 400 d
Event Archive...
Decide
Decide
Rule evaluation
event plus context
Rule evaluation...
Notification intent
notif.intent
Notification intent...
Notification Store
Postgres · 30 d hot
Notification Store...
Resolve identity
Resolve identity
Recipient Profile
subject_ref to contact
Recipient Profile...
PII Vault
encrypted · erasable
PII Vault...
Rendered message
memory only · never logged
Rendered message...
Deliver and record
Deliver and record
Channel workers
Channel workers
Delivery Store
ClickHouse · 90 d
Delivery Store...
Audit Log
MinIO WORM · 7 y
Audit Log...
malformed · pipeline unblocked
malformed · pipeline unblocked
5 min micro-batch
5 min micro-batch
on decision
on decision
gRPC · scoped read
gRPC · scoped read
per attempt
per attempt
audit
audit
Data Flow — Event, Notification, Delivery
Data Flow — Event, Notification, Delivery
External / third party
External / third party
Application we own
Application we own
Decision point
Decision point
Queue / topic
Queue / topic
Data store
Data store
Security / platform
Security / platform
failure / alternate
failure / alternate
batch
batch
synchronous
synchronous
event / async
event / async
PII enters at stage five and never reaches the bus, the archive or a log line. An erasure request purges the vault; the event and notification records survive with an unresolvable reference.
PII enters at stage five and never reaches the bus, the archive or a log line. An erasure request purges the vault; the event and notification records survive with an unresolvable reference.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Data Flow Where data comes from, what it becomes at each stage, and the one stage where PII appears. HTML page SVG draw.io
10
Event topic
Event topic
Rule evaluation
Rule evaluation
Intent topic
Intent topic
Channel topic
Channel topic
Failure path
Failure path
P0 · transactional
P0 · transactional
evt.valid.p0
48 part · 7 d
evt.valid.p0...
Reserved slot group
never starved
Reserved slot group...
notif.intent.p0
24 partitions
notif.intent.p0...
deliver.*.p0
consumed first
deliver.*.p0...
dlq.p0
page on first message
dlq.p0...
P1 · operational
P1 · operational
evt.valid.p1
120 part · 7 d
evt.valid.p1...
Default slot group
Default slot group
notif.intent.p1
60 partitions
notif.intent.p1...
deliver.*.p1
deliver.*.p1
dlq.p1
alert above 100
dlq.p1...
P2 · bulk and marketing
P2 · bulk and marketing
evt.valid.p2
72 part · 3 d
evt.valid.p2...
Capped parallelism
sheds first
Capped parallelism...
notif.intent.p2
36 partitions
notif.intent.p2...
deliver.*.p2
throttled to spare capacity
deliver.*.p2...
dlq.p2
daily digest
dlq.p2...
Replay and recovery
Replay and recovery
Event Archive
Iceberg · queryable
Event Archive...
Replay Service
range plus filter
Replay Service...
evt.replay
separate consumer group
evt.replay...
Dedup suppresses
dedup_key match
Dedup suppresses...
Replay audit
who · range · count
Replay audit...
Event Topology — Priority Tiers, DLQ and Replay
Event Topology — Priority Tiers, DLQ and Replay
Queue / topic
Queue / topic
Application we own
Application we own
Data store
Data store
Decision point
Decision point
Security / platform
Security / platform
Partition key is tenant_id:entity_id, so OrderCreated to OrderDelivered stay ordered for one order without any global ordering guarantee. Tier separation is what stops a marketing burst delaying a one-time passcode.
Partition key is tenant_id:entity_id, so OrderCreated to OrderDelivered stay ordered for one order without any global ordering guarantee. Tier separation is what stops a marketing burst delaying a one-time passcode.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Event Topology and Priority Tiers Which topics exist, why three tiers, how ordering is guaranteed, and how a stuck message gets out. HTML page SVG draw.io
11
Authoritative — cannot be rebuilt, must be backed up
Authoritative — cannot be rebuilt, must be backed up
Configuration
Configuration
Config Store
rules · tenants · providers
Config Store...
Template versions
every version kept
Template versions...
Consent and identity
Consent and identity
Preference Store
Postgres · RLS
Preference Store...
PII Vault
pgcrypto · per-tenant DEK
PII Vault...
Audit Log
MinIO object lock · 7 y
Audit Log...
Operational — rebuildable from the event log
Operational — rebuildable from the event log
Hot
Hot
Notification Store
partitioned · 30 d
Notification Store...
Redis
dedup 7 d · counters
Redis...
Kafka topics
3 to 7 d retention
Kafka topics...
Analytical
Analytical
Delivery Store
ClickHouse · 90 d raw
Delivery Store...
Rolled-up aggregates
13 months
Rolled-up aggregates...
Archive — immutable, cheap, queryable
Archive — immutable, cheap, queryable
Long term
Long term
Event Archive
Iceberg · 400 d
Event Archive...
Notification archive
Iceberg · 13 months
Notification archive...
Backup target
cross-region · PITR 7 d
Backup target...
Erasure job
purge cascade within 30 d
Erasure job...
Trino
archive queries · read only
Trino...
5 min micro-batch
5 min micro-batch
nightly at 30 d
nightly at 30 d
hourly
hourly
hard delete
hard delete
continuous WAL
continuous WAL
SQL read
SQL read
Data Architecture and Storage Zones
Data Architecture and Storage Zones
Data store
Data store
Security / platform
Security / platform
Queue / topic
Queue / topic
Application we own
Application we own
batch
batch
failure / alternate
failure / alternate
event / async
event / async
synchronous
synchronous
The retention numbers are the design. Anything in the operational zone can be lost and reconstructed from the archive; nothing in the authoritative zone can, which is what backup and DR are sized around.
The retention numbers are the design. Anything in the operational zone can be lost and reconstructed from the archive; nothing in the authoritative zone can, which is what backup and DR are sized around.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Data Architecture and Retention What is authoritative, what is rebuildable, and how long each thing lives. HTML page SVG draw.io
12
tenant
tenant_id PK
name
tier
quota_events_sec
home_region
tenant...
event
event_id PK
tenant_id FK
event_type
source_app
subject_ref
correlation_id
priority
idempotency_key UQ
occurred_at
event...
notification_rule
rule_id PK
tenant_id FK
event_type
condition_cel
channels
enabled
version
notification_rule...
template
template_id PK
tenant_id FK
code
channel
state
template...
recipient
recipient_id PK
tenant_id FK
subject_ref UQ
locale
timezone
consent_state
recipient...
notification
notification_id PK
event_id FK
rule_id FK
recipient_id FK
channel
state
dedup_key UQ
template_version_id FK
scheduled_for
notification...
schedule
schedule_id PK
notification_id FK
fire_at
cron_expr
digest_window
schedule...
template_version
version_id PK
template_id FK
semver
locale
body_ref
published_at
template_version...
user_preference
preference_id PK
recipient_id FK
category
channel
enabled
quiet_start
quiet_end
frequency_cap
user_preference...
delivery_attempt
attempt_id PK
notification_id FK
provider_id FK
attempt_no
status
provider_ref
error_class
latency_ms
attempted_at
delivery_attempt...
channel_provider
provider_id PK
tenant_id FK
channel
vendor
rank
credential_path
tps_limit
channel_provider...
audit_record
audit_id PK
tenant_id FK
actor
action
target_ref
before_after
occurred_at
audit_record...
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
1 : 0..1
1 : 0..1
1 : 1
1 : 1
1 : N
1 : N
1 : N
1 : N
1 : N
1 : N
Core Data Model
Core Data Model
Three domains, deliberately separate: event is what happened, notification is what the platform decided, delivery_attempt is what happened while trying. Every table carries tenant_id and is protected by row-level security; those relationships are omitted for readability.
Three domains, deliberately separate: event is what happened, notification is what the platform decided, delivery_attempt is what happened while trying. Every table carries tenant_id and is protected by row-level security; those relationships are omitted for readability.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Core Data Model The entities a decision reads and writes, and how they join. HTML page SVG draw.io
13
Consent basis
Consent basis
Preference respected
Preference respected
Quiet hours
Quiet hours
Frequency cap
Frequency cap
Override rule
Override rule
Security · P0
Security · P0
Legitimate interest
no opt-in needed
Legitimate interest...
Channel order only
cannot disable
Channel order only...
Ignored
sent immediately
Ignored...
None
None
Always overrides
logged as override
Always overrides...
Transactional · P0
Transactional · P0
Contract
service delivery
Contract...
Channel choice
at least one mandatory
Channel choice...
Ignored
Ignored
None
None
Overrides opt-out
reason recorded
Overrides opt-out...
Operational · P1
Operational · P1
Legitimate interest
Legitimate interest
Fully respected
per category
Fully respected...
Deferred
recipient timezone
Deferred...
10 per day
then digested
10 per day...
No override
No override
Marketing · P2
Marketing · P2
Explicit opt-in
timestamp and source
Explicit opt-in...
Fully respected
Fully respected
Deferred
plus local send window
Deferred...
3 per week
hard cap
3 per week...
Never overrides
regulatory breach
Never overrides...
Digest · P2
Digest · P2
Inherits source class
Inherits source class
Digest opt-in
replaces per-event
Digest opt-in...
Scheduled
recipient chosen hour
Scheduled...
1 per window
1 per window
P0 items break out
sent separately
P0 items break out...
Preference, Consent and Override Policy
Preference, Consent and Override Policy
Security / platform
Security / platform
Application we own
Application we own
Decision point
Decision point
Risk / gap
Risk / gap
The override column is the whole point of this view. Anything that overrides a stated preference writes an audit record naming the rule that authorised it, so a regulator can be answered from one query.
The override column is the whole point of this view. Anything that overrides a stated preference writes an audit record naming the rule that authorised it, so a regulator can be answered from one query.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Preference, Consent and Override Policy Which notifications may ignore a stated preference, and on what authority. HTML page SVG draw.io

4 · Runtime

The decision an event triggers, the path a message takes, when it is sent, and what happens when a provider fails.
14
Should we notify?
Should we notify?
Validated event
evt.valid.p1
Validated event...
Rule match
CEL over event plus context
Rule match...
Suppression list
incident mute · tenant pause
Suppression list...
Who?
Who?
Recipient resolution
subject_ref to recipients
Recipient resolution...
Fan-out
1 event to N notifications
Fan-out...
What and where?
What and where?
Preference check
category · consent
Preference check...
Channel selection
ordered · fallback chain
Channel selection...
Template bind
code · locale · version pinned
Template bind...
When?
When?
Timing router
now · delay · cron · digest
Timing router...
Quiet-hours defer
recipient timezone
Quiet-hours defer...
Guardrails
Guardrails
Deduplication
dedup_key · 24 h window
Deduplication...
Frequency cap
per category per window
Frequency cap...
Rate limit
tenant · user · provider
Rate limit...
Suppressed
no rule · opted out · capped
Suppressed...
Dispatch
Dispatch
Notification record
CREATED to QUEUED
Notification record...
Channel topic
deliver.channel.tier
Channel topic...
duplicate
duplicate
over cap
over cap
next allowed window
next allowed window
Notification Orchestration — From Event to Dispatch Decision
Notification Orchestration — From Event to Dispatch Decision
Queue / topic
Queue / topic
Decision point
Decision point
Application we own
Application we own
Risk / gap
Risk / gap
Data store
Data store
failure / alternate
failure / alternate
event / async
event / async
Every suppression increments a labelled counter and writes a reason on the notification record. A notification that was never sent is still a notification with an answer, which is what makes the support question answerable.
Every suppression increments a labelled counter and writes a reason on the notification record. A notification that was never sent is still a notification with an answer, which is what makes the support question answerable.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Notification Orchestration The seven questions asked of every event, in order, and what happens when the answer is no. HTML page SVG draw.io
15
Order Service
Order Service
Event Ingest API
Event Ingest API
Kafka
Kafka
Rules Engine
Rules Engine
Orchestrator
Orchestrator
Render Service
Render Service
Email Worker
Email Worker
Postal MTA
Postal MTA
1. POST /v1/events OrderCreated
1. POST /v1/events OrderCreated
2. schema · idempotency_key · event_id
2. schema · idempotency_key · event_id
3. produce acks=all
3. produce acks=all
4. 202 event_id · p95 80 ms
4. 202 event_id · p95 80 ms
5. consume evt.valid.p1
5. consume evt.valid.p1
6. CEL match · fan-out to 3 channels
6. CEL match · fan-out to 3 channels
7. notif.intent
7. notif.intent
8. preferences · quiet hours · dedup
8. preferences · quiet hours · dedup
9. render(template v2.4, en-GB)
9. render(template v2.4, en-GB)
10. resolve subject_ref to address
10. resolve subject_ref to address
11. subject · html · text
11. subject · html · text
12. deliver.email.p1
12. deliver.email.p1
13. consume · claim token
13. consume · claim token
14. SMTP submit
14. SMTP submit
15. 250 queued · message-id
15. 250 queued · message-id
16. receipt SENT
16. receipt SENT
17. webhook DELIVERED
17. webhook DELIVERED
OrderCreated to Delivered Email — Critical Flow
OrderCreated to Delivered Email — Critical Flow
Happy path only. Retry, provider failover and dead-lettering are on view 17. Contact details are resolved inside the Render Service and never travel on the bus.
Happy path only. Retry, provider failover and dead-lettering are on view 17. Contact details are resolved inside the Render Service and never travel on the bus.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Event to Delivered Email The happy path end to end, with the latency budget at each hop. HTML page SVG draw.io
16
Timing decision
Timing decision
Durable wait
Durable wait
Aggregate
Aggregate
Becomes due
Becomes due
Dispatch
Dispatch
Immediate · 85%
Immediate · 85%
Send now
no preference to defer
Send now...
None
straight through
None...
notif.due
p95 under 400 ms
notif.due...
Channel topic
Channel topic
Delayed
Delayed
Delay t
rule sets the offset
Delay t...
Temporal timer
survives restart
Temporal timer...
notif.due
timer fires
notif.due...
Channel topic
Channel topic
Scheduled
Scheduled
At schedule
cron per tenant
At schedule...
Temporal cron
tenant timezone
Temporal cron...
notif.due
timer fires
notif.due...
Channel topic
Channel topic
Quiet hours
Quiet hours
In quiet window
recipient timezone
In quiet window...
Defer to next hour
Defer to next hour
Jittered release
spread over 15 min
Jittered release...
notif.due
window opens
notif.due...
Channel topic
Channel topic
Digest
Digest
Digest opted in
replaces per-event
Digest opted in...
Session window
Flink · 1 h default
Session window...
Aggregate buffer
Redis roll-up
Aggregate buffer...
Digest render
N items to 1
Digest render...
notif.due
one message
notif.due...
Channel topic
Channel topic
Breakout
Breakout
P0 inside a window
security or transactional
P0 inside a window...
None
window bypassed
None...
Left in the digest
would delay a passcode
Left in the digest...
notif.due
immediately
notif.due...
Channel topic
Channel topic
Timing — Immediate, Delayed, Scheduled and Digested
Timing — Immediate, Delayed, Scheduled and Digested
Decision point
Decision point
Application we own
Application we own
Queue / topic
Queue / topic
Data store
Data store
Risk / gap
Risk / gap
Every lane converges on the same notif.due stream, so the dispatcher has one code path. Timing state is durable in Temporal and Kafka, never an in-process timer — a pod restart during a two-day delay loses nothing.
Every lane converges on the same notif.due stream, so the dispatcher has one code path. Timing state is durable in Temporal and Kafka, never an in-process timer — a pod restart during a two-day delay loses nothing.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Timing, Scheduling and Digests Six ways a notification can be timed, and the one due-work stream they all converge on. HTML page SVG draw.io
17
Email Worker
Email Worker
Rate Limiter
Rate Limiter
Postal MTA
Postal MTA
Amazon SES
Amazon SES
Delivery Store
Delivery Store
Kafka DLQ
Kafka DLQ
Alertmanager
Alertmanager
1. acquire token
1. acquire token
2. granted
2. granted
3. submit · attempt 1
3. submit · attempt 1
4. 451 transient
4. 451 transient
5. attempt 1 · TRANSIENT
5. attempt 1 · TRANSIENT
6. backoff 2^n plus jitter
6. backoff 2^n plus jitter
7. submit · attempt 2
7. submit · attempt 2
8. timeout
8. timeout
9. circuit opens · 5 fails in 30 s
9. circuit opens · 5 fails in 30 s
10. failover submit · attempt 3
10. failover submit · attempt 3
11. 250 accepted
11. 250 accepted
12. attempt 3 · SENT via secondary
12. attempt 3 · SENT via secondary
13. half-open probe after 60 s
13. half-open probe after 60 s
14. 550 permanent · mailbox unknown
14. 550 permanent · mailbox unknown
15. park with full context
15. park with full context
16. DLQ depth breach
16. DLQ depth breach
Delivery Failure — Retry, Failover and Dead-Lettering
Delivery Failure — Retry, Failover and Dead-Lettering
Transient failures retry then fail over; permanent failures go straight to the DLQ without spending the retry budget. Five attempts over 30 minutes for P1, two over 60 seconds for P0.
Transient failures retry then fail over; permanent failures go straight to the DLQ without spending the retry budget. Five attempts over 30 minutes for P1, two over 60 seconds for P0.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Retry, Failover and Dead-Lettering What happens when a provider says no, and how a message that cannot be sent stays recoverable. HTML page SVG draw.io

5 · Operations

Where it runs, how a change reaches production, how it scales, what is watched, and how an operator answers a question.
18
Primary region · eu-west-1 · active
Primary region · eu-west-1 · active
Zone A
Zone A
Kubernetes nodes
8 × 16 vCPU
Kubernetes nodes...
Kafka brokers
4 × NVMe
Kafka brokers...
PostgreSQL primary
CloudNativePG
PostgreSQL primary...
ClickHouse shard 1
ClickHouse shard 1
Zone B
Zone B
Kubernetes nodes
8 × 16 vCPU
Kubernetes nodes...
Kafka brokers
4 × NVMe
Kafka brokers...
PostgreSQL sync replica
quorum commit
PostgreSQL sync replica...
ClickHouse shard 2
ClickHouse shard 2
Zone C
Zone C
Kubernetes nodes
6 × 16 vCPU
Kubernetes nodes...
Kafka brokers
4 × NVMe
Kafka brokers...
PostgreSQL async replica
read scale
PostgreSQL async replica...
ClickHouse shard 3
ClickHouse shard 3
DR region · eu-central-1 · warm standby
DR region · eu-central-1 · warm standby
Standby footprint
Standby footprint
Kubernetes
30% · scales on promote
Kubernetes...
Kafka
MirrorMaker 2 target
Kafka...
PostgreSQL
streaming replica · lag under 5 s
PostgreSQL...
MinIO
cross-region replication
MinIO...
Vault replica
performance standby
Vault replica...
Global load balancer
health-based failover
Global load balancer...
Harbor registry
geo-replicated
Harbor registry...
Channel providers
region-agnostic endpoints
Channel providers...
primary traffic
primary traffic
on promotion · RTO 30 min
on promotion · RTO 30 min
MirrorMaker 2
MirrorMaker 2
streaming replication
streaming replication
synchronous commit
synchronous commit
Deployment and Failure Domains
Deployment and Failure Domains
Security / platform
Security / platform
Queue / topic
Queue / topic
Data store
Data store
Interface / broker
Interface / broker
External / third party
External / third party
synchronous
synchronous
failure / alternate
failure / alternate
event / async
event / async
Kafka with replication factor 3 and min.insync.replicas 2 gives RPO 0 in-region for an acknowledged event. Losing one zone costs capacity, not data; losing the region costs at most five minutes of unreplicated events.
Kafka with replication factor 3 and min.insync.replicas 2 gives RPO 0 in-region for an acknowledged event. Losing one zone costs capacity, not data; losing the region costs at most five minutes of unreplicated events.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Deployment and Failure Domains What runs where, what survives a zone loss, and what a region loss actually costs. HTML page SVG draw.io
19
Source
Source
GitLab monorepo
trunk plus MR
GitLab monorepo...
Config repository
rules · templates · quotas
Config repository...
Build
Build
GitLab CI
build · unit · lint
GitLab CI...
Container image
distroless · reproducible
Container image...
SBOM and signature
Syft · Cosign
SBOM and signature...
Gates
Gates
Unit and contract
coverage 80%
Unit and contract...
Vulnerability scan
no critical
Vulnerability scan...
Policy check
OPA Gatekeeper
Policy check...
Schema compatibility
Avro backward
Schema compatibility...
Non-production
Non-production
dev
auto on merge
dev...
staging
load test 25 k events/s
staging...
Chaos run
broker kill · provider outage
Chaos run...
Approval
Approval
Change gate
2 approvals · CAB for schema
Change gate...
Argo CD
GitOps sync · drift alarm
Argo CD...
Production
Production
Argo Rollouts
canary 5% to 100%
Argo Rollouts...
SLO watch
burn rate under 2×
SLO watch...
Automatic rollback
on breach within 10 min
Automatic rollback...
blocked
blocked
breach
breach
config only
config only
Delivery Pipeline and Environments
Delivery Pipeline and Environments
Interface / broker
Interface / broker
Application we own
Application we own
Security / platform
Security / platform
Decision point
Decision point
failure / alternate
failure / alternate
event / async
event / async
Rules and templates ship through the same pipeline as code, so a rule change is versioned, reviewed and revertible. Config-only changes skip the image build but not the gates.
Rules and templates ship through the same pipeline as code, so a rule change is versioned, reviewed and revertible. Config-only changes skip the image build but not the gates.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Delivery Pipeline and Environments How a change reaches production, and what stops a bad one. HTML page SVG draw.io
20
Pressure signal
Pressure signal
Consumer lag
per topic per tier
Consumer lag...
Due-work depth
scheduled backlog
Due-work depth...
Provider latency
p95 per vendor
Provider latency...
Controller
Controller
KEDA scaler
lag-driven
KEDA scaler...
HPA
CPU for stateless APIs
HPA...
Flink autoscaler
reactive parallelism
Flink autoscaler...
Scaled unit
Scaled unit
Ingest API
12 to 90 pods
Ingest API...
Render Service
20 to 160 pods
Render Service...
Channel workers
80 to 600 pods
Channel workers...
Flink task managers
24 to 120
Flink task managers...
Hard ceiling
Hard ceiling
Partition count
concurrency ceiling
Partition count...
Provider TPS
contracted, not elastic
Provider TPS...
Database connections
PgBouncer pool
Database connections...
Backpressure
Backpressure
Shed P2 first
marketing pauses
Shed P2 first...
429 with Retry-After
at the gateway
429 with Retry-After...
Park to DLQ
last resort, replayable
Park to DLQ...
ceiling reached
ceiling reached
vendor cap
vendor cap
still over budget
still over budget
Autoscaling, Capacity and Backpressure
Autoscaling, Capacity and Backpressure
Application we own
Application we own
Security / platform
Security / platform
Queue / topic
Queue / topic
External / third party
External / third party
Data store
Data store
Decision point
Decision point
Interface / broker
Interface / broker
failure / alternate
failure / alternate
Capacity target: 5 000 events/s normal, 25 000 peak, fan-out 1.8, so 45 000 notifications/s peak and roughly 750 M a day. Scaling stops at the partition count, which is why partitions are provisioned for peak and not for today.
Capacity target: 5 000 events/s normal, 25 000 peak, fan-out 1.8, so 45 000 notifications/s peak and roughly 750 M a day. Scaling stops at the partition count, which is why partitions are provisioned for peak and not for today.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Autoscaling, Capacity and Backpressure What each tier scales on, where scaling stops, and what the platform does once it has stopped. HTML page SVG draw.io
21
Emit
Emit
Collect
Collect
Store
Store
Consume
Consume
Act
Act
Metrics
Metrics
OpenTelemetry SDK
RED per service
OpenTelemetry SDK...
OTel Collector
OTel Collector
Prometheus and Mimir
13 months
Prometheus and Mimir...
SLO dashboard
Grafana
SLO dashboard...
Page on burn rate
Alertmanager
Page on burn rate...
Logs
Logs
Structured JSON
event_id · correlation_id
Structured JSON...
Fluentd
PII redaction filter
Fluentd...
Loki
30 d
Loki...
Incident search
Incident search
Runbook link
Runbook link
Traces
Traces
W3C tracecontext
propagated on the bus
W3C tracecontext...
OTel Collector
tail sampling · errors kept
OTel Collector...
Tempo
7 d
Tempo...
Event to provider span
one trace end to end
Event to provider span...
Latency budget alert
Latency budget alert
Notification lifecycle
Notification lifecycle
State transitions
CREATED to DELIVERED
State transitions...
Receipt topic
Receipt topic
ClickHouse
90 d attempts
ClickHouse...
Notification timeline
one query per recipient
Notification timeline...
Support answer
target under 2 min
Support answer...
Queue and backlog
Queue and backlog
Consumer lag
per tier per tenant
Consumer lag...
Kafka exporter
Kafka exporter
Prometheus
Prometheus
Backlog board
DLQ depth included
Backlog board...
KEDA scale-out
before the page
KEDA scale-out...
Tenant and cost
Tenant and cost
Per-tenant counters
sent · failed · suppressed
Per-tenant counters...
OTel Collector
OTel Collector
ClickHouse
ClickHouse
Chargeback report
provider spend per tenant
Chargeback report...
Quota enforcement
Quota enforcement
Observability — Signals, Storage and Who Gets Woken
Observability — Signals, Storage and Who Gets Woken
Application we own
Application we own
Interface / broker
Interface / broker
Data store
Data store
Security / platform
Security / platform
Queue / topic
Queue / topic
The fourth row is the one the requirement asks for by name. Because event_id, notification_id and provider_ref are on the same trace and the same ClickHouse row, why customer X did not receive notification Y is one query, not four systems.
The fourth row is the one the requirement asks for by name. Because event_id, notification_id and provider_ref are on the same trace and the same ClickHouse row, why customer X did not receive notification Y is one query, not four systems.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Observability and Operations Every signal, where it goes, and how a support engineer answers the question in one place. HTML page SVG draw.io
22
Read — any operator with tenant scope
Read — any operator with tenant scope
Inspect
Inspect
Event browser
by id · type · window
Event browser...
Notification history
per recipient timeline
Notification history...
Failure explorer
error class · provider
Failure explorer...
System health
lag · DLQ · SLO
System health...
Configure — tenant administrator, reviewed change
Configure — tenant administrator, reviewed change
Business configuration
Business configuration
Rule management
CEL editor · dry run
Rule management...
Template management
draft · publish · roll back
Template management...
Preference defaults
Preference defaults
Platform configuration
Platform configuration
Provider management
rank · credentials · TPS
Provider management...
Retry policy
attempts · backoff · budget
Retry policy...
Feature flags
Unleash · per tenant
Feature flags...
Tenant quotas
Tenant quotas
Dangerous — step-up authentication, four-eyes, always audited
Dangerous — step-up authentication, four-eyes, always audited
Recovery actions
Recovery actions
Event replay
range capped · second approver
Event replay...
DLQ redrive
batch limited
DLQ redrive...
PII erasure
irreversible
PII erasure...
Tenant pause
stops all sends
Tenant pause...
Admin API
everything the console does
Admin API...
Keycloak
roles · step-up MFA
Keycloak...
Audit Log
WORM · 7 y
Audit Log...
roles
roles
who · range · count
who · range · count
subject · scope
subject · scope
diff
diff
Administration and Operations Surface
Administration and Operations Surface
Application we own
Application we own
Security / platform
Security / platform
Interface / broker
Interface / broker
Risk / gap
Risk / gap
synchronous
synchronous
event / async
event / async
The console is a client of the Admin API and has no privileged path of its own. The red row is the reason this view exists: replay at scale can duplicate millions of notifications, so it is gated, capped and audited rather than merely permitted.
The console is a client of the Admin API and has no privileged path of its own. The red row is the reason this view exists: replay at scale can duplicate millions of notifications, so it is gated, capped and audited rather than merely permitted.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Administration and Operations Surface What an operator can do, who may do it, and which actions are dangerous enough to gate. HTML page SVG draw.io

6 · Assurance

Trust boundaries, the secretless producer path, and every failure mode with the cover for it.
24
Untrusted · internet
Untrusted · internet
Producer application
Producer application
Recipient device
Recipient device
Attacker
forged and replayed events
Attacker...
Perimeter · DMZ
Perimeter · DMZ
WAF and DDoS
per-tenant rps
WAF and DDoS...
Envoy Gateway
TLS 1.3 · mTLS
Envoy Gateway...
Token validation
Keycloak JWKS
Token validation...
Application · private
Application · private
Ingest API
tenant claim to topic ACL
Ingest API...
Decision plane
Istio mTLS · no egress
Decision plane...
Delivery workers
egress proxy only
Delivery workers...
Data · restricted
Data · restricted
Apache Kafka
SASL · ACL per tenant
Apache Kafka...
PostgreSQL
RLS · encrypted volumes
PostgreSQL...
PII Vault
per-tenant data key
PII Vault...
Audit Log
WORM · 7 y
Audit Log...
Identity and secrets
Identity and secrets
Keycloak
OIDC · client credentials
Keycloak...
HashiCorp Vault
1 h dynamic leases
HashiCorp Vault...
SPIRE
workload SVID
SPIRE...
HTTPS · mTLS
HTTPS · mTLS
replay rejected
replay rejected
WSS
WSS
TLS 1.3
TLS 1.3
JWT verify
JWT verify
tenant claim
tenant claim
SASL · ACL
SASL · ACL
mTLS · RLS
mTLS · RLS
scoped read
scoped read
SVID
SVID
Security Architecture — Trust Zones
Security Architecture — Trust Zones
External / third party
External / third party
Person or role
Person or role
Risk / gap
Risk / gap
Security / platform
Security / platform
Interface / broker
Interface / broker
Application we own
Application we own
Queue / topic
Queue / topic
Data store
Data store
synchronous
synchronous
failure / alternate
failure / alternate
event / async
event / async
Producers never hold a provider credential — the platform leases them per tenant from Vault. Events carry a subject reference rather than PII, so a compromised producer token leaks routing metadata and not contact data.
Producers never hold a provider credential — the platform leases them per tenant from Vault. Events carry a subject reference rather than PII, so a compromised producer token leaks routing metadata and not contact data.
v 1.0 · owner Security Architecture · date 2026-08
v 1.0 · owner Security Architecture · date 2026-08
Text is not SVG - cannot display
Security Architecture — Trust Zones Where the boundaries are, what crosses them, and where an attacker actually arrives. HTML page SVG draw.io
25
Producer workload
Producer workload
SPIRE Agent
SPIRE Agent
Keycloak
Keycloak
Envoy Gateway
Envoy Gateway
Ingest API
Ingest API
Kafka
Kafka
Vault
Vault
1. attest workload
1. attest workload
2. X.509 SVID · 1 h
2. X.509 SVID · 1 h
3. client_credentials grant
3. client_credentials grant
4. JWT · tenant_id · scopes · 15 min
4. JWT · tenant_id · scopes · 15 min
5. POST /v1/events · mTLS plus bearer
5. POST /v1/events · mTLS plus bearer
6. SPIFFE ID against tenant allow-list
6. SPIFFE ID against tenant allow-list
7. JWKS · cached 10 min
7. JWKS · cached 10 min
8. forward with verified claims
8. forward with verified claims
9. scope events:write · tenant match
9. scope events:write · tenant match
10. lease database credential · 1 h
10. lease database credential · 1 h
11. dynamic credential
11. dynamic credential
12. produce as tenant principal
12. produce as tenant principal
13. ACL denied · wrong tenant topic
13. ACL denied · wrong tenant topic
14. 202 Accepted · event_id
14. 202 Accepted · event_id
Identity and Access — Producer Authentication
Identity and Access — Producer Authentication
No long-lived secret is issued to a producer. The mTLS identity proves the workload, the JWT carries the tenant, and Kafka enforces the same tenant a second time so a bug in the API cannot cross a tenant boundary.
No long-lived secret is issued to a producer. The mTLS identity proves the workload, the JWT carries the tenant, and Kafka enforces the same tenant a second time so a bug in the API cannot cross a tenant boundary.
v 1.0 · owner Security Architecture · date 2026-08
v 1.0 · owner Security Architecture · date 2026-08
Text is not SVG - cannot display
Identity and Access — Producer Authentication How a producing workload proves who it is without ever holding a long-lived secret. HTML page SVG draw.io
26
Infrastructure failure — red is the failure, the label is the cover
Infrastructure failure — red is the failure, the label is the cover
Compute and storage
Compute and storage
Node loss
PDB plus reschedule
Node loss...
Broker loss
RF 3 · min.insync 2
Broker loss...
Zone loss
3 zones · quorum holds
Zone loss...
Region loss
warm DR · RTO 30 min
Region loss...
Stateful services
Stateful services
PostgreSQL primary loss
auto failover under 30 s
PostgreSQL primary loss...
Flink job failure
checkpoint restore 60 s
Flink job failure...
Temporal outage
immediate path unaffected
Temporal outage...
Redis loss
dedup fails open · duplicates over loss
Redis loss...
Dependency failure
Dependency failure
Providers
Providers
Provider outage
circuit opens · secondary
Provider outage...
Both vendors down
park in DLQ · replay
Both vendors down...
Provider throttling
adaptive TPS · P2 shed
Provider throttling...
Internal services
Internal services
Template Service down
last published cached
Template Service down...
Config Service down
rules held in Flink state
Config Service down...
Customer Master down
24 h profile cache
Customer Master down...
Poison message
3 strikes then DLQ
Poison message...
Recovery tooling
Replay Service · DLQ console
Recovery tooling...
Backup
PITR 7 d · cross-region
Backup...
Game day
quarterly · these exact modes
Game day...
redrive when healthy
redrive when healthy
inspect and fix
inspect and fix
restore path
restore path
exercises quarterly
exercises quarterly
Failure Modes and Disaster Recovery
Failure Modes and Disaster Recovery
Risk / gap
Risk / gap
Application we own
Application we own
Data store
Data store
Security / platform
Security / platform
failure / alternate
failure / alternate
batch
batch
RPO 0 in-region for an acknowledged event, RPO 5 min cross-region, RTO 30 min. The one accepted trade-off is drawn in the second box: if Redis is lost, deduplication fails open and the platform sends duplicates rather than dropping notifications.
RPO 0 in-region for an acknowledged event, RPO 5 min cross-region, RTO 30 min. The one accepted trade-off is drawn in the second box: if Redis is lost, deduplication fails open and the platform sends duplicates rather than dropping notifications.
v 1.0 · owner SRE and Security Architecture · date 2026-08
v 1.0 · owner SRE and Security Architecture · date 2026-08
Text is not SVG - cannot display
Failure Modes and Disaster Recovery Everything that can break, what covers it, and the one trade-off deliberately accepted. HTML page SVG draw.io
Open svg/<view>.svg or drawio/<view>.drawio in draw.io Desktop or at app.diagrams.net to edit. The SVG carries the diagram inside it, so it is both the picture and the source. This folder is self-contained — copy it whole and every link still resolves.