Customer 360 & Real-Time Risk Intelligence Platform · View 06 of 20

Real-Time Event Ingestion

The path a high-volume event takes from producer to enriched topic, including contract validation, dead-lettering, deduplication and windowed feature computation.

Editable source SVG draw.io All views

Volume and sizing

Design point: 40k events per second peak, 1.2 billion events per day
Partitioned by customer key so per-customer ordering is preserved
7 days hot retention on brokers, longer history in tiered object storage

Delivery semantics

Kafka delivers at-least-once; deduplication on event key gives effective exactly-once
Lakehouse sinks use transactional writes so a retry cannot double-count
Deduplication state is held for 24 hours, covering realistic producer retry windows

Failure handling

Contract violations go to a dead-letter topic, never silently dropped
Dead-letter records keep the original payload and are replayable after a fix
Consumer lag above threshold pages the on-call before SLA is breached