AI Code Completion Service  ·  View 15 of 21  ·  Runtime

Four Fates of One Keystroke

Five request populations across the same six stages. Two of them cost nothing; one costs most of the fleet.

Editable source SVG draw.io All views
Arrive Affinity lookup Admission Batch slot Prefix state Outcome Warm session — 85% Session key known Replica hit Queue wait Joins in-flight batch HBM cache hit p50 120 ms Cold session — 15% First keystroke No replica held Any replica accepted Admitted, lower priority Recompute prefix +180 ms at p95 Drained mid-session Node upgrade or spot loss Affinity invalidated Re-placed, not errored New replica NVMe tier read Degrades, never fails Refused at the door Peak, 90 k rps Lookup skipped Deadline unreachable Empty, 0 GPU seconds Cancelled — 55% of all Developer kept typing Same replica Already admitted Sequence evicted Prefix kept warm Reclaimed in 50 ms Inference Plane — Four Fates of One Keystroke Row four costs nothing and row five costs most of the fleet. Budgeting for row five is what makes row one affordable. v 1.0 · owner Developer Experience Architecture · date 2026-10

Why this view exists

  • The cancelled row is 55% of all traffic. Budgeting for it explicitly — eager reclamation, prefix kept warm, wasted generation as a capped metric — is what makes the warm row affordable.
  • The drained row is the cost of session affinity, and it is a latency penalty rather than an error. A design that errored here would have made affinity unacceptable.
  • The refused row costs zero accelerator seconds, which is the entire argument for admission control over queueing.

Assumptions

  • Warm 85% / cold 15%; cold costs ≤ 180 ms extra at p95.
  • Cancellation releases accelerator work within 50 ms at p99; wasted generation ≤ 18% of accelerator seconds.
  • Continuous batching to 64 concurrent sequences per replica; fleet utilisation ≥ 55% over a working day.

Risks

  • The cold row and the drained row share a mechanism, so a bad drain protocol shows up as a latency regression that looks like a model problem.
◀  Context Assembly All views Model Promotion and Gating  ▶
Solution Architecture v1.0 · Microsoft Azure · Developer Experience Architecture · 2026-10 · 21 views · 15 architecture decision records15-four-fates-of-a-request