AI Code Completion Service  ·  View 07 of 21  ·  Structure

Layered Architecture

Eight layers, of which exactly one holds durable truth — and it is not on the request path.

Editable source SVG draw.io All views
Editor clients VS Code reference client JetBrains Neovim Debounce & cancel Local redaction Edge Front Door anycast + WAF Seat check Tenant quota Geography pin Client version gate Context FIM assembly Open-tab sources Repo retrieval Token budget Stable-prefix order Admission Model choice Affinity lookup Redis Deadline admission Ordered shedding Inference Continuous batching AKS GPU pools KV cache — HBM KV cache — DRAM KV cache — NVMe Speculative decode Post-process Stop-sequence trim Suffix dedup Secret scan fail closed Public-code match Degeneracy check Measurement Event Hubs Privacy scrub Data Explorer Experiment arms Promotion gate Control plane Policy & seats Azure SQL Model registry ACR, by digest Flags & kill switch Key Vault Azure Monitor suggest deadline admit stream events promote cached policy Layered Architecture External / third party Application we own Interface / broker Security / platform Decision point Data store Queue / topic synchronous event / async The control plane spans every layer and sits on none of them: it is read through a cache and never on the critical path. v 1.0 · owner Developer Experience Architecture · date 2026-10

Decisions

  • The control plane spans every layer and sits on none: it is read through a bounded-staleness cache so a control-plane outage degrades policy freshness rather than stopping suggestions.
  • The KV prefix cache is drawn as three tiers because the tier a session lands in is the difference between 120 ms and 300 ms, which makes it an architectural fact rather than an implementation detail.
  • Measurement is a layer, not an aspect. In a product with no labels, the telemetry path is as load-bearing as the inference path.

Assumptions

  • Prefix-cache hit rate ≥ 85% on consecutive requests within a session.
  • Context assembly p95 ≤ 25 ms on the client, ≤ 15 ms server-side.

Risks

  • Eight layers is eight hops of fixed overhead inside a 300 ms budget; any layer that grows a synchronous dependency spends budget that cannot be recovered downstream.
◀  Journey — Switching On a Tenant All views Platform Components  ▶
Solution Architecture v1.0 · Microsoft Azure · Developer Experience Architecture · 2026-10 · 21 views · 15 architecture decision records07-layered-architecture