AI Code Completion Service  ·  View 18 of 21  ·  Operations

Observability

Six signal classes against six stages, and the one row with no error signal behind it.

Editable source SVG draw.io All views
Editor client Edge Context Admission Inference Filters & telemetry Latency TTFT, client measured Edge overhead Assembly p95 Queue wait p99 Tokens / second Filter time p99 Saturation In-flight per session Quota rejections Budget truncations Admission refusals Batch occupancy Event Hubs lag Cache & affinity Prefix stability Affinity hit rate Tier hit: HBM / DRAM / NVMe Quality Displayed, accepted Recipe version Per-model acceptance Retention at 120 s Safety Client version floor Redaction hits Degeneracy ejections Secret suppressions Cost Requests per seat Tokens per prompt Wasted generation % GPU seconds per tenant Cost per accepted Observability — Signal by Stage Only one row has no error signal behind it: Quality. That is why the control arm is held back permanently rather than per experiment. v 1.0 · owner Developer Experience Architecture · date 2026-10

The empty-signal row

  • Quality is the only row where failure produces no error, no alert and no page: a model that has quietly become less useful looks exactly like a healthy fleet.
  • That is why a control arm is held back permanently rather than only during an experiment, and why retention is measured continuously rather than at promotion.

Decisions

  • Latency is measured client-side, from debounce fire to ghost text painted. Server-side latency is a diagnostic, not the contract.
  • Admission refusals and wasted-generation percentage are first-class operational signals, because they are the two that trade directly against cost.
  • Measurement failures are routed separately from reliability alerts: an on-call engineer must be able to tell a broken exporter from a broken fleet.

Risks

  • Client-measured latency depends on a client the platform does not fully control; a plugin release can move the headline metric without anything server-side changing.
◀  Deployment Topology All views Model and Quality Lifecycle  ▶
Solution Architecture v1.0 · Microsoft Azure · Developer Experience Architecture · 2026-10 · 21 views · 15 architecture decision records18-observability