Real-Time Analytics Platform  ·  View 07 of 21  ·  Data

Event Data Flow

Where data comes from, what format and cadence it moves in, what is retained at each hop, and where an event goes when it cannot be processed.

Editable source SVG draw.io All views
Source
Source
Change Rows
Avro · row image
Change Rows...
Application Events
JSON · 1.1 KB avg
Application Events...
Device Telemetry
protobuf · 240 B
Device Telemetry...
Capture and validate
Capture and validate
Idempotency Key
event_id + payload hash
Idempotency Key...
Event Collector
schema validation
Event Collector...
Rejected Events
invalid schema
Rejected Events...
Durable log
Durable log
Bronze Capture
Parquet · 7 y · immutable
Bronze Capture...
Azure Event Hubs
7 d · 256 partitions
Azure Event Hubs...
Dead-Letter Hub
14 d
Dead-Letter Hub...
Process and enrich
Process and enrich
Dimension Join
broadcast Delta
Dimension Join...
Windowed Aggregation
tumbling · sliding · session
Windowed Aggregation...
Watermark Gate
lateness 2 min
Watermark Gate...
Hot store
Hot store
Silver & Gold Delta
ADLS · rebuildable
Silver & Gold Delta...
ADX Raw Table
31 d hot · 400 d total
ADX Raw Table...
Materialized Views
1 min · 5 min · 1 h
Materialized Views...
Serve
Serve
Gold Event Topics
enriched · async
Gold Event Topics...
Query API
SQL and KQL
Query API...
Dashboards
Power BI · Grafana
Dashboards...
schema fail
schema fail
Capture 15 min
Capture 15 min
unprocessable
unprocessable
replay
replay
sub-second reads
sub-second reads
Event Data Flow — Format, Cadence and Retention
Event Data Flow — Format, Cadence and Retention
External / third party
External / third party
Application we own
Application we own
Risk / gap
Risk / gap
Data store
Data store
Queue / topic
Queue / topic
Decision point
Decision point
Interface / broker
Interface / broker
failure / alternate
failure / alternate
batch
batch
synchronous
synchronous
Bronze is the only zone that cannot be rebuilt; everything downstream is derivable from it.
Bronze is the only zone that cannot be rebuilt; everything downstream is derivable from it.
v 1.0 · owner Data Architecture · date 2026-08
v 1.0 · owner Data Architecture · date 2026-08
Text is not SVG - cannot display

Retention by zone

  • Event Hubs 7 days · dead-letter 14 days · ADX hot cache 31 days, total 400 days
  • Bronze capture 7 years in ADLS, immutable with a legal-hold policy available
  • Silver and gold are rebuildable and carry no independent retention obligation

No silent loss

  • An event is either committed to Event Hubs or the producer receives a non-2xx response
  • Schema rejects and unprocessable events land in a dead-letter hub with the failure reason
  • Dead-letter depth and schema-reject rate are first-class alerting signals, not log lines

Deliberate omissions

  • Window semantics and late-arrival behaviour are drawn in the windowing view
  • Tier transitions and lifecycle policies are drawn in the storage view
  • Per-interface authentication is drawn in the security and identity views