Real-Time Analytics Platform · View 11 of 21 · Runtime

Stream Processing Pipeline

What each micro-batch does, stage by stage, and exactly where the delivery guarantee is established.

Editable source SVG draw.io All views

Exactly-once, precisely

Transport is at-least-once; sinks are idempotent, giving effectively-once end to end
ADX deduplicates on an ingest-by tag; Delta writes carry txnAppId and txnVersion
The checkpoint advances only after every sink acknowledges, so a crash replays, never skips

Backpressure

maxOffsetsPerTrigger caps each batch so a burst lengthens the queue instead of the batch
Event Hubs retention is the buffer: 7 days of absorption before anything is at risk
Autoscale triggers on consumer lag, not on CPU, because lag is the signal that matters

Failure handling

Undecodable and unprocessable events are quarantined with the failure reason attached
A dead-letter event is never silently dropped and is replayable after the fix
State store corruption is recovered by replay from bronze, not by partial checkpoint repair

◀ Schema Contract Governance All views Windowing and Late-Arriving Events ▶