LLM Rate Limiting & Traffic Management Service  ·  View 20 of 24  ·  Operations

Observability and SLOs

Every signal the brief asks for, placed against the stage that emits it, with the SLO that makes it actionable.

Editable source SVG draw.io All views
Edge & admission
Edge & admission
Decision
Decision
Coordination
Coordination
Egress & providers
Egress & providers
Usage & cost
Usage & cost
Metrics
Metrics
Throughput
rps · shed · queue
Throughput...
Decision latency
p50 / p99 · verdicts
Decision latency...
Shard health
Lua p99 · lease hit
Shard health...
Provider health
latency · breaker · 429
Provider health...
Consumption
tokens/min · burn rate
Consumption...
Logs
Logs
Access log
no request body
Access log...
Decision audit
1% + all rejects
Decision audit...
Script errors
failover events
Script errors...
Provider errors
bodies redacted
Provider errors...
Ledger gaps
reconciliation
Ledger gaps...
Traces
Traces
Ingress span
Ingress span
Check span
scope resolution
Check span...
Valkey span
one per lease refill
Valkey span...
Upstream span
provider call
Upstream span...
Commit span
linked by request_id
Commit span...
SLOs & alerts
SLOs & alerts
Availability 99.99%
52 min / year
Availability 99.99%...
p99 < 10 ms
budget 4.3 min/mo
p99 < 10 ms...
Shard p99 < 1.5 ms
lease hit ≥ 90%
Shard p99 < 1.5 ms...
Failover < 5 s
trip to reroute
Failover < 5 s...
Staleness < 60 s
ledger lag < 5 min
Staleness < 60 s...
Observability — signal type against pipeline stage
Observability — signal type against pipeline stage
Every signal is emitted through OpenTelemetry and correlated by request_id, which is the same identifier the reservation ledger and the usage row carry.
Every signal is emitted through OpenTelemetry and correlated by request_id, which is the same identifier the reservation ledger and the usage row carry.
v 1.0 · owner Data & AI Global Practice
v 1.0 · owner Data & AI Global Practice
Text is not SVG - cannot display

Decisions

  • Everything is correlated by request_id — the same identifier the reservation holds, the usage row carries and the trace links. One identifier answers 'what happened to this request' across all five stages.
  • Rejections are logged in full; allows are sampled at 1%. Rejections are what a tenant will dispute, and they are a small fraction of traffic, so full retention is affordable.
  • Lease hit rate is treated as a first-class SLI, not a debug metric. It is the leading indicator for both latency and Valkey load, and it degrades before the p99 does.

The SLOs

  • Availability 99.99% (52 min/year) · decision p99 < 10 ms with a 4.3 min/month error budget.
  • Valkey script p99 < 1.5 ms · lease hit rate ≥ 90% · provider failover < 5 s.
  • Policy staleness < 60 s · usage ledger lag < 5 min.

Privacy constraint

  • Access logs record no request body. Provider error responses are redacted before they reach Loki, because a provider error can echo prompt content.
  • Traces carry scope identifiers and token counts as attributes, never message content.
  • Tenant-facing dashboards are scoped by ClickHouse row policies, so a tenant cannot query another tenant's consumption.