Real-Time Analytics Platform  ·  View 18 of 21  ·  Operations

Observability and Service Level Objectives

Which signal is collected at which stage, and which of them will page somebody — the matrix an SRE reads before accepting the service.

Editable source SVG draw.io All views
Ingestion
Ingestion
Event log
Event log
Processing
Processing
Hot store
Hot store
Serving
Serving
Latency and lag
Latency and lag
Accept p95 · 200 ms
Accept p95 · 200 ms
Consumer lag
Consumer lag
Batch duration
Batch duration
Ingestion latency
Ingestion latency
Query p95 · 2 s
Query p95 · 2 s
Throughput
Throughput
Events accepted/s
Events accepted/s
Throughput units
Throughput units
Rows per batch
Rows per batch
Rows ingested/min
Rows ingested/min
Concurrent queries
Concurrent queries
Errors
Errors
4xx / 5xx rate
4xx / 5xx rate
Quota throttles
Quota throttles
Failed batches
Failed batches
Ingestion failures
Ingestion failures
Query timeouts
Query timeouts
Data quality
Data quality
Schema rejects
Schema rejects
Dead-letter depth
Dead-letter depth
Late and dropped events
Late and dropped events
Count reconciliation
Count reconciliation
Freshness watermark
Freshness watermark
Alerting
Alerting
Accept-rate burn alert
Accept-rate burn alert
Lag over 60 s pages
Lag over 60 s pages
Checkpoint stall pages
Checkpoint stall pages
Freshness SLO burn
Freshness SLO burn
Query SLO burn
Query SLO burn
Observability and Service Level Objectives
Observability and Service Level Objectives
Security / platform
Security / platform
Risk / gap
Risk / gap
Application we own
Application we own
Queue / topic
Queue / topic
Every event carries event_id and trace context, so a single record is traceable from SDK to dashboard.
Every event carries event_id and trace context, so a single record is traceable from SDK to dashboard.
v 1.0 · owner SRE · date 2026-08
v 1.0 · owner SRE · date 2026-08Text is not SVG - cannot display

The four SLOs

  • Freshness: 95% of events queryable within 5 seconds of acceptance
  • Query latency: 95% of standard dashboard queries under 2 seconds
  • Availability: 99.9% successful event acceptance and successful query response
  • Completeness: zero unexplained gaps between events accepted and events queryable

End-to-end tracing

  • event_id and W3C trace context travel from SDK through to the hot store row
  • A single event can be located at every hop, which is what makes completeness auditable
  • Reconciliation compares accepted counts against ingested counts on a 5-minute cadence

Alerting discipline

  • Pages fire on burn rate against an SLO, never on a raw threshold crossing
  • Dead-letter depth and watermark skew are treated as data-loss precursors, not warnings
  • Everything else is a dashboard signal and a ticket, not a page
◀  CI/CD and Environment Promotion All views Reliability and Recovery Loop  ▶
Solution Architecture v1.0 · Azure · Data & AI Architecture · 2026-0818-observability-and-slo