# Event-Driven Notification Platform

**Solution Architecture v1.0 · Data & AI Global Practice · 2026-08 · 26 views · open-source stack**

A multi-tenant platform that accepts business events from any internal producer and decides what to notify, to whom, on which channel and when — then composes, dispatches and tracks the result across six channels. Producers publish facts; they know nothing about templates, channels or providers.

Six decisions carry the design:

1. **Events and notifications are separate domains** with separate storage and separate retention — event is what happened, notification is what the platform decided (including deciding to send nothing), delivery attempt is what happened while trying.
2. **Rules are evaluated once, centrally, in a stateful stream job** (Flink with broadcast CEL rules), not by per-event RPC to a rule service and not inside the channel workers.
3. **Durable workflow state is used only where it is needed** — Temporal carries delay, scheduling, digests and channel fallback for roughly 15 percent of traffic; the immediate path is pure Kafka, which keeps Temporal off the critical path.
4. **Three priority tiers run end to end** — separate topics, consumer groups and Flink slot groups — so a 20 M-message campaign cannot delay a one-time passcode. Priority inside a single topic cannot fix head-of-line blocking.
5. **PII is resolved at render time and never touches the bus or the archive.** Events carry a subject reference; a compromised producer token leaks routing metadata, not contact data.
6. **At-least-once with idempotent consumers is the stated contract**, with deduplication at three points that *fails open* — under a Redis loss the platform sends duplicates rather than dropping notifications.

Everything is open source except the channel providers themselves, where FCM and APNs have no self-hosted equivalent: Kafka (Strimzi), Flink, Temporal, PostgreSQL (CloudNativePG), ClickHouse, Redis, MinIO with Apache Iceberg, Keycloak, HashiCorp Vault, SPIRE, Envoy Gateway, Istio, Kubernetes, Argo CD, GitLab CI, OPA Gatekeeper, OpenTelemetry and the Grafana LGTM stack.

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The view index — 26 views grouped into six acts, every format linked |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning it deliberately omits, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same 26 views as SVG with the diagram XML embedded — re-opens in diagrams.net fully editable |
| `diagrams/drawio/*.drawio` | draw.io native source |
| `specs/views.json` | Diagram specifications — the source of truth (assembled from `part-a..f.json`) |
| `specs/manifest.json` | Acts, page titles, subtitles and the decision / assumption / risk cards (assembled from `manifest-a..c.json`) |
| `scripts/build.sh` | Rebuilds every deliverable from the specs |
| `scripts/pin-icons.mjs` | Pins one mark per component so the same box carries the same icon on every page |
| `ask.md` | The original requirement |

Every component carries its own mark — 464 icons embedded in the files, so the deliverable renders identically anywhere with no external requests. Each view page links to its own SVG and draw.io source, back to the index, and to the previous and next view.

To rebuild after editing a spec:

```bash
bash scripts/build.sh          # requires Node 20+ and draw.io Desktop
```

Edit `specs/part-a.json` … `part-f.json` and `specs/manifest-a.json` … `manifest-c.json`; the build script assembles `views.json` and `manifest.json` from them and fails if the two sets of view ids disagree.

---

## The six acts

| Act | What it lands | Views |
|---|---|---|
| 1 · Context and scope | The boundary, who publishes, who receives, what the platform never owns | 01–02 |
| 2 · Structure | The layering rule, the components, every interface, the channel contract, content governance, and what tenants share | 03–08 |
| 3 · Data | Three separate domains, where each lives, how long, and what consent permits | 09–13 |
| 4 · Runtime | The decision an event triggers, the path a message takes, when it sends, and what happens when a provider fails | 14–17 |
| 5 · Operations | Where it runs, how a change ships, how it scales, what is watched, and how an operator answers a question | 18–23 |
| 6 · Assurance | Trust boundaries, the secretless producer path, and every failure mode with the cover for it | 24–26 |

---

## The 26 views

| # | View | Layout | What it answers |
|---|---|---|---|
| 01 | System Context | context | Who publishes events, who receives notifications, what the platform never owns |
| 02 | High-Level Architecture | flow | The shape of the platform in one picture |
| 03 | Layered Architecture | bands | What depends on what, and the one dependency pointing the wrong way |
| 04 | Container and Component Architecture | nested | Every deployable unit, its technology, its replica floor, the protocol on every hop |
| 05 | Integration and Interface Catalogue | hub | Every way something enters or leaves, with protocol and cadence |
| 06 | Channel Abstraction | swimlane | The five-step contract every channel implements, and what genuinely differs |
| 07 | Template and Content Lifecycle | flow | How a message goes from draft to published, and how a bad one is taken back |
| 08 | Multi-Tenancy and Noisy-Neighbour Control | nested | What tenants share, what each gets alone, what stops one hurting the rest |
| 09 | Data Flow | flow | What data becomes at each stage, and the one stage where PII appears |
| 10 | Event Topology and Priority Tiers | swimlane | Which topics exist, why three tiers, how ordering holds, how a stuck message gets out |
| 11 | Data Architecture and Retention | nested | What is authoritative, what is rebuildable, how long each thing lives |
| 12 | Core Data Model | er | The entities a decision reads and writes, and how they join |
| 13 | Preference, Consent and Override Policy | grid | Which notifications may ignore a stated preference, and on what authority |
| 14 | Notification Orchestration | flow | The seven questions asked of every event, and what happens when the answer is no |
| 15 | Event to Delivered Email | sequence | The happy path end to end, with the latency budget at each hop |
| 16 | Timing, Scheduling and Digests | swimlane | Six ways a notification can be timed, and the one stream they converge on |
| 17 | Retry, Failover and Dead-Lettering | sequence | What happens when a provider says no, and how it stays recoverable |
| 18 | Deployment and Failure Domains | nested | What runs where, what survives a zone loss, what a region loss costs |
| 19 | Delivery Pipeline and Environments | flow | How a change reaches production, and what stops a bad one |
| 20 | Autoscaling, Capacity and Backpressure | flow | What scales on what, where scaling stops, what happens once it has |
| 21 | Observability and Operations | grid | Every signal, where it goes, and how a support engineer answers in one place |
| 22 | Administration and Operations Surface | nested | What an operator can do, who may do it, which actions are gated |
| 23 | The Operator Loop | cycle | How an alert becomes an answer, a fix and a clean backlog |
| 24 | Security Architecture — Trust Zones | zones | Where the boundaries are, what crosses them, where an attacker arrives |
| 25 | Identity and Access — Producer Authentication | sequence | How a workload proves who it is without holding a long-lived secret |
| 26 | Failure Modes and Disaster Recovery | nested | Everything that can break, what covers it, the one trade-off accepted |

---

## Assumptions this design was built on

The requirement lists ten questions to answer before designing. None had answers, so each was assumed explicitly and carried onto the relevant view's cards. **Every one of these changes the design if the real answer differs.**

| Question | Assumed | Effect if wrong |
|---|---|---|
| Events/sec and notifications/sec | 5 000 normal, 25 000 peak; fan-out 1.8, so 45 000 notifications/s and ~750 M a day | Partition counts, ClickHouse sizing and the whole tiering argument scale with this |
| Channels required initially | Email, SMS, push, in-app, webhook, chat. Voice deferred | Voice fits the existing adapter contract; adding it is one adapter |
| Single- or multi-tenant | Multi-tenant | Single-tenant removes roughly a third of the design — views 08 and most of 24 |
| Delivery guarantee | At-least-once with idempotent consumers and best-effort suppression | Effectively-once would need a transactional outbox per channel and a much slower path |
| Notification latency SLA | p95 under 2 s event to provider accepted for transactional | A sub-second target removes Flink checkpointing from the transactional path |
| Per-entity ordering | Required per `tenant_id:entity_id`; no global ordering | Global ordering forces a single partition and caps throughput at one consumer |
| Scheduling, digests, batching | All required | Without them, Temporal and the digest windows come out entirely |
| Rule complexity | Declarative CEL conditions, no arbitrary code | Arbitrary logic becomes a webhook callout — a different and slower design |
| RPO / RTO and regions | RPO 0 in-region, 5 min cross-region; RTO 30 min; single active region | An active-active requirement forces conflict resolution on notification state |
| Compliance | GDPR and CCPA; audit 7 years; PII deletable | HIPAA or PCI would move the PII vault into a separately certified boundary |

Retention figures (Kafka 3–7 days, events 400 days, notifications 30 days hot then 13 months, delivery attempts 90 days, audit 7 years) are defensible defaults, not contractual numbers — confirm them against the tenant agreement before build.

---

## What still needs a decision

| Item | Why it matters | Owner |
|---|---|---|
| Dedicated Flink jobs for the top five tenants | One shared job means a hot tenant key can lag a partition and delay unrelated tenants. Key salting fixes the bulk tier only, because salting the transactional tier breaks per-entity ordering | Platform engineering — before go-live |
| Second push vendor | FCM and APNs have no alternative. The only fallback is degrading to SMS or in-app, which changes the notification, not just the route | Product |
| Quiet-hours release jitter | Timezone-aligned quiet-hours windows create a synchronised release at the boundary. A 15-minute jitter is designed in but not yet load-tested | SRE |
| Native Kafka producer path | Producers that write straight to the broker bypass gateway quota and schema enforcement, leaving only broker ACLs and the registry. Either accept the weaker path or require the HTTP API | Enterprise architecture |
| Legacy producers without mTLS | Scoped API keys are a documented lower trust level and need an agreed expiry date | Security architecture |