AI Code Completion Service  ·  View 13 of 21  ·  Runtime

Critical Flow — One Keystroke

Nineteen messages, of which the tenth is the one that keeps the latency contract.

Editable source SVG draw.io All views
Developer Editor plugin Edge + seat gate Prompt assembler Affinity router GPU replica Safety filters Telemetry 1. keystroke 2. cancel previous in-flight 3. debounce 60 ms 4. POST /suggest (deadline 2000 ms) 5. seat, quota, version, geography 6. forward, deadline 1940 ms 7. FIM assembly, budget, stable order 8. prompt + session key 9. affinity lookup; admit or refuse 10. no capacity in deadline — empty 11. admit to in-flight batch 12. prefix cache hit (85%) 13. first tokens, streaming 14. trim, suffix dedup, secret scan 15. displayable at 120 ms p50 16. ghost text painted 17. Tab, or keeps typing 18. accepted / cancelled 19. retention probe at 120 s Critical Flow — One Keystroke, 300 Milliseconds Message 10 is the one that keeps the contract: the refusal is issued before any accelerator work starts. v 1.0 · owner Developer Experience Architecture · date 2026-10

Read message 10 first

  • The refusal is issued before any accelerator work starts. That single ordering decision is what makes a 4× burst a capacity question rather than a latency incident.
  • Messages 2 and 3 are in the client because only the editor knows the keystroke: cancellation and debounce cannot be server-side at this budget.

Assumptions

  • The edge spends ≤ 60 ms of the 2,000 ms client deadline on identity, quota, version and geography.
  • Prefix-cache hit on consecutive keystrokes in a session: 85%.
  • Retention is probed at 120 s after acceptance, asynchronously, from the client.

Risks

  • The self-messages hide the real tail: assembly, affinity lookup and filtering are each small, and together they are a third of the p95 budget before a token is generated.
◀  Data Model All views Context Assembly  ▶
Solution Architecture v1.0 · Microsoft Azure · Developer Experience Architecture · 2026-10 · 21 views · 15 architecture decision records13-keystroke-to-ghost-text