Architecture One-Pager

AI Code Completion Service · Solution Architecture v1.0 · Microsoft Azure · Developer Experience Architecture · 2026-10 · 21 views · 15 architecture decision records

The unit of capacity is a warm session, not a request: inference is session-affine to the replica holding that editor's KV prefix, and every other tier is stateless and disposable.

Everyone has met this system from the outside. You type three characters, grey text appears ahead of the cursor, and you either press Tab or keep typing and watch it vanish. It is the most-used AI product in the world by interaction count, and it is the only one that fires on a keystroke rather than on a prompt. That one fact rearranges every assumption a normal inference service is built on. The user's patience is not measured in seconds but in a couple of hundred milliseconds, because the alternative to a suggestion is not a worse suggestion — it is the developer having already typed the line themselves, at which point an arriving suggestion costs attention and returns nothing. More than half of all generated work is thrown away before anyone reads it, so the economics are set by cancelled requests rather than by completed ones. There are no labels: nobody tells the platform whether a suggestion was good, and the only evidence available is whether a developer took it and whether it survived the afternoon. And the input is the most sensitive artefact the customer owns, which means an enterprise's security reviewer holds a veto that no latency number will overturn. A design that treats this as "an LLM behind an API" gets all four of those wrong at once.

Make the warm session the unit the fleet is organised around, and make everything else abandonable. A request arrives at a global edge that spends sixty milliseconds on identity, seat, quota, client version and geography, then stamps a deadline that travels with it. Context is assembled against a declared token budget from the current file, open tabs, recent edits and permission-scoped repository retrieval, ordered so that the prompt's leading segment is stable across consecutive keystrokes. An affinity router asks which accelerator replica already holds that session's key-value prefix and places the request there; an admission gate refuses outright anything that cannot make its deadline, before a single accelerator second is spent. Generation streams back through trimming, suffix deduplication and a fail-closed secret scan, and lands as grey text. Everything durable sits beside that path and never on it: policy and seats in a strongly consistent store read through a thirty-second cache, models in an immutable content-addressed registry, and interaction telemetry in an append-only log whose code-bearing stream is separated at the boundary and, for enterprise tenants, does not exist. Under any failure the correct output is a fast, silent nothing.

What it is, and what it is not

• A placement problem over replicas holding warm prefixes — not A stateless inference pool behind a load balancer
• A service whose correct failure output is silence — not A service that returns an error the developer can see
• Work budgeted on the assumption that most of it is discarded — not Work priced per completed request
• Accepted-character retention measured against a held-back control arm — not An acceptance rate on a dashboard
• A zero-retention promise made by the absence of a path — not A retention setting switched off
• Repository permission resolved per request — not An index built once under the permissions of the day
• Filters that each declare whether they fail open or closed — not A filter chain that times out
• Cost expressed per accepted suggestion — not Cost expressed per request served
• Three regions, each a complete working set — not One global fleet with regional front doors

The decisions that are the architecture

1. The warm session is the unit of capacity (ADR-01) — Inference is session-affine to the replica holding that editor's prefix, which makes the inference tier a placement problem rather than a balancing one and makes scale-in correctness-adjacent. It is the one decision every later view either honours or contradicts.
2. The prefix cache is tiered and partitioned by tenant (ADR-02) — Device memory, then host memory, then node-local NVMe — and a hard tenant boundary, because a content-keyed shared cache would turn the fleet's chief efficiency mechanism into its chief isolation risk.
3. Prompt order is a contract, not a style (ADR-03) — The leading segment of the prompt must be stable across consecutive keystrokes or the cache is invalidated on every one of them. Volatile context goes last, and the ordering rule is versioned and logged.
4. Refuse before generating; never return late (ADR-04) — Admission control evaluates the remaining deadline against queue depth and rejects at the door for zero accelerator cost. A 4× burst becomes a capacity question instead of a latency incident.
5. The untrusted client owns debounce and cancellation (ADR-05) — Only the editor knows the keystroke, so suppression, debounce and cancel-on-keystroke live there — while no policy decision, filter or quota is enforced only there.
6. Shed by refusing, in a declared order (ADR-06) — Multi-line before single-line, speculative before user-initiated, lower tier before higher. Degradation is a published decision rather than a latency curve nobody chose.
7. Retention, not acceptance, is the quality metric (ADR-07) — Accepted characters still present after 120 seconds. A more confident model raises acceptance and can lower retention, and acceptance alone cannot see the difference.
8. Offline evaluation can only refuse (ADR-08) — The held-out suite is a promotion gate, never a verdict: its correlation with retention is too weak to put a model in front of 1.2 million developers.
9. A control arm is held back permanently (ADR-09) — Silent quality regression is the only failure class in this system with no error signal. The answer is a standing control arm and automated rollback, not an alert.
10. Zero retention by absence, not by setting (ADR-10) — For an enterprise tenant the retaining path does not exist — including in logs, traces and crash dumps. A reviewer is right to discount a configuration claim, and this is not one.
11. Every filter declares its failure direction (ADR-11) — The secret scan is fail-closed inside an 8 ms budget; the public-code filter is fail-closed for tenants that chose to block. Which way a filter fails is a product decision, never an implementation accident.
12. Repository permission is resolved per request (ADR-12) — Against the requesting principal, at request time — not cached from index build. An index that outlives a permission change must not serve it.
13. Cost is measured per accepted suggestion (ADR-13) — Per-request cost hides the 55% that is cancelled, which is where the money actually goes. Wasted generation is a capped, budgeted metric with an owner.
14. Two models, routed by request shape (ADR-14) — A fast single-line model and an escalated multi-line model. Deeper cascades pay twice at the tail and cannot be evaluated, because the counterfactual suggestion is never generated.
15. A region is a complete working set (ADR-15) — Prefix caches are never replicated across regions. Losing a region costs cold starts rather than correctness, and the fleet-fragmentation cost is accepted as the price of the latency contract.

Why this should still be right in ten years

The model, the serving runtime, the accelerator, the editor and the cloud will all be replaced inside this platform's life. These are the properties that should outlast them.

• Locality beats statelessness when the state is expensive to rebuild. "Route to where the warm state already is" is not a property of today's attention implementation. Any generative decoder will keep per-sequence state that is cheaper to reuse than to recompute, and any architecture serving an incremental, high-frequency editing signal will find that the previous request's state is almost the whole of the next one's. The specific mechanism — key-value cache, compiled state, a learned summary — will change. The decision that clients are placed rather than balanced survives all of them.
• A deadline that travels with the request outlives the service that honours it. Deadline propagation plus refuse-rather-than-queue is a protocol property, not a framework feature. It is what makes capacity planning honest: load shows up as refusals you can count rather than as a latency distribution that degrades silently. This pattern was right before this product existed and will be right after it.
• Behavioural measurement is the only thing that scales with model capability. Every offline benchmark eventually saturates or is trained against. What does not saturate is whether a human kept the output. Retention against a held-back control arm will still be the right instrument when the model writes whole functions, because the question it answers — did this survive contact with the work — does not depend on what the model can do.
• Privacy by absence survives audits that privacy by configuration does not. A control implemented as a missing code path cannot be misconfigured, cannot drift, and does not need to be re-attested after every refactor. As regulation around training data and source-code provenance tightens, the designs that will still be adoptable are the ones where the sensitive path was never built.
• Unit economics expressed in the user's unit, not the machine's. Cost per accepted suggestion will outlive the accelerator generation, the model size and the pricing model, because it is denominated in the thing the customer is buying. Cost per request, cost per token and cost per GPU-hour all stop being comparable the moment any of those three change.

Non-functional targets

Every figure below is a stated assumption. They are listed with the mechanism that is supposed to deliver them and the view where that mechanism is drawn, so a reviewer can disagree with a number and follow it straight to the design that depends on it.

Scope

In scope

• The editor-facing request lifecycle: a streaming, cancellable fill-in-the-middle suggestion API with client-supplied deadlines, and the published client contract (debounce, single in-flight per session, cancel-on-keystroke, suppression rules, version floor)
• Context assembly from the open workspace: current file, open tabs, recent edits, imported symbols and permission-scoped repository retrieval, under an enforced token budget with priority truncation and stable-prefix ordering
• Low-latency inference serving: continuous batching, a tiered key-value prefix cache, session-affine placement, deadline-aware admission control and an ordered shedding ladder
• Model portfolio and routing: an immutable content-addressed registry, request-shape routing between a fast and an escalated model, traffic splitting, canary gating and ten-minute rollback
• Suggestion safety: secret scanning, public-code match filtering with warn and block modes, suffix deduplication and degenerate-output detection, each with a declared failure direction
• Measurement: display/cancel/accept/retain telemetry, accepted-character retention, randomised online experiments with stable assignment, and a promotion gate with automated rollback
• Tenancy, privacy and administration: zero-retention mode, geography pinning, seats, policy with declared precedence, consent and audit, and per-tenant cost attribution

Explicitly out of scope

• Training, fine-tuning or adapting the models — there is deliberately no data path from tenant code to training
• The chat and agent surfaces that sit beside completion in the same editor: they share an editor and almost nothing else, with a latency budget measured in seconds
• The editor plugins themselves, beyond the contract they must honour and the version gate that enforces it
• Code review, static analysis and security scanning of the developer's own code
• Billing and invoicing: usage figures are exported nightly to a billing platform that owns the money
• Per-repository model adaptation, next-edit prediction and an on-device fallback model, all named in phase 3 of the requirement and deliberately deferred

What a four-week prototype should prove

The prototype's job is to falsify the three central claims — that session affinity is worth the operational cost it imposes, that refusing at the door really does hold the latency contract under burst, and that retention is measurable well enough to gate a rollout — on one language and a few hundred developers, not to build a platform.

1. One editor (VS Code), one language, one model, one region, 200 volunteer developers, with client-measured first-token latency as the headline number from day one
2. Session-affine routing against a measured baseline of stateless routing on the same hardware, reporting the p95 difference and the cost of a deliberate mid-session drain
3. A deadline-aware admission gate under a synthetic 4× burst, proving that refusals rise while p95 for admitted requests stays inside budget — and counting the accelerator seconds the refusals saved
4. Stable-prefix prompt ordering against a deliberately volatile ordering, to put a real number on the cache-invalidation cost the ordering rule exists to prevent
5. Accepted-character retention instrumented end to end, including the 120-second probe, with a held-back control arm large enough to detect a 1.5% relative acceptance change
6. A zero-retention tenant with an independent review of logs, traces and crash dumps for code-bearing content — the one test whose failure invalidates the adoption argument rather than the performance one

• A GPU replica holding warm sessions is drained under live traffic: every affected session continues on another replica with a measured latency penalty and zero errors, and the penalty is within the 180 ms cold-prefix budget rather than a full recompute
• A synthetic 4× burst is applied: admission refusals rise, p95 first-token latency for admitted requests does not move, and the accelerator seconds spent on refused requests is zero — which is the entire argument for refusing over queueing
• The prompt recipe is switched to a deliberately volatile ordering with everything else held constant: the prefix hit rate must collapse and first-token p95 must rise by roughly the cold-prefix penalty, confirming that prompt order is a latency decision and not a style one
• A developer's repository access is revoked while they are mid-session: the next keystroke's retrieval must no longer return content from that repository, without waiting for an index rebuild
• The secret scanner is taken offline: no suggestion is displayed to any tenant, the suppression is counted, and the outage appears as a product outage rather than as a quiet stream of unfiltered completions
• A zero-retention tenant is driven through a forced crash, a trace capture and a full debug-logging cycle: an independent reviewer searching every sink finds no code-bearing content, which is the test whose failure invalidates the adoption argument rather than the performance one

Open risks, carried rather than hidden

The reasoning behind every component and technology choice is in the Architecture Decision Record: 15 records across 5 areas, each with the alternatives that lost and what the choice costs.