Customer 360 & Real-Time Risk Intelligence Platform  ·  View 04 of 20

Container Architecture (C4 Level 2)

The deployable units inside the platform boundary, the technology each one runs on, and the protocols between them.

Editable source SVG draw.io All views
Customer 360 Platform — Cloud Landing Zone
Customer 360 Platform — Cloud Landing Zone
Ingestion Segment
Ingestion Segment
Kafka Connect Cluster
Debezium, S3 sink
Kafka Connect Cluster...
Kafka Brokers
36 brokers, RF 3
Kafka Brokers...
Schema Registry
Avro, BACKWARD
Schema Registry...
Event Gateway
REST proxy, mTLS
Event Gateway...
Processing Segment
Processing Segment
Flink Job Manager
Enrichment topology
Flink Job Manager...
Spark Streaming Jobs
Bronze & silver writer
Spark Streaming Jobs...
Airflow Scheduler
Celery workers
Airflow Scheduler...
Spark Batch Jobs
Gold builds, backfill
Spark Batch Jobs...
Storage & Serving Segment
Storage & Serving Segment
Object Store Buckets
Bronze / silver / gold
Object Store Buckets...
Delta Tables
ACID, time travel
Delta Tables...
SQL Warehouse
Serverless endpoints
SQL Warehouse...
Profile API Service
Containers, autoscaled
Profile API Service...
Profile Store
Low-latency key value
Profile Store...
Governance & Platform Segment
Governance & Platform Segment
Unity Catalog
Grants, lineage
Unity Catalog...
Business Glossary
Terms, ownership
Business Glossary...
Tokenisation Vault
PII surrogates
Tokenisation Vault...
Observability Stack
Metrics, logs, traces
Observability Stack...
Source Systems
Core, cards, CRM, digital
Source Systems...
Consumers
Fraud, BI, ML, regulators
Consumers...
CDC
CDC
consume
consume
write bronze
write bronze
JDBC
JDBC
Container Architecture (C4 Level 2)
Container Architecture (C4 Level 2)
Interface / broker
Interface / broker
Queue / topic
Queue / topic
Application we own
Application we own
Data store
Data store
Security / platform
Security / platform
External / third party
External / third party
event / async
event / async
synchronous
synchronous
batch
batch
Cross-cutting grants, encryption and lineage apply to every segment; drawn once in the governance box.
Cross-cutting grants, encryption and lineage apply to every segment; drawn once in the governance box.
v 1.0 · owner Enterprise Data Architecture · date 2026-08
v 1.0 · owner Enterprise Data Architecture · date 2026-08Text is not SVG - cannot display

Runtime choices

  • Kafka for the event backbone; Kafka Connect hosts Debezium and sink connectors
  • Flink for stateful enrichment; Spark Structured Streaming for lakehouse writes
  • Airflow for batch, quality and backfill orchestration only — never for streaming

Why two processing engines

  • Flink carries keyed state and event-time joins that Spark micro-batching handles poorly
  • Spark owns the lakehouse write path where Delta transactions and compaction live
  • The split is by responsibility, not preference; each has one job it is best at

State and scaling

  • Flink task managers autoscale on consumer lag; state is checkpointed to object storage
  • Spark clusters mix spot and on-demand, with on-demand floors for SLA-bound jobs
  • Profile API is stateless and horizontally scaled behind the gateway
Solution Architecture v1.0 · Enterprise Data Architecture · 2026-0804-container-architecture