# AI Code Completion Service

**Solution Architecture v1.0 · Microsoft Azure · Developer Experience Architecture · 2026-10 · 21 views · 15 architecture decision records**

You type three characters and grey text appears ahead of the cursor. You press Tab, or you keep typing and it vanishes. It is the most-used AI product in the world by interaction count — it fires on a keystroke, not on a prompt — and the one where patience is measured in a couple of hundred milliseconds, because the alternative to a suggestion is not a worse suggestion but the developer having already typed the line. This package is the architecture behind that grey text: the editor-facing request lifecycle, context assembly from the open workspace, low-latency inference with prefix-cache reuse, model routing, suggestion filtering, the telemetry that is this product's only quality signal, and the tenancy and privacy controls that let an enterprise switch it on. The assumed operating context is 1.2 million daily active developers across 40 enterprise tenants, 2.4 billion suggestion requests a day at 35,000 per second steady and 90,000 at the working-day peak, of which at least 55% are cancelled before anything is displayed, served from roughly 4,000 accelerators across three regions — built on AKS GPU node pools with a tiered key-value prefix cache, Container Apps for every stateless tier, Front Door at the edge, Azure SQL for policy and seats, Container Registry for models addressed by digest, and Event Hubs into Data Explorer for measurement.

The product is not the suggestion. It is **a suggestion that arrives before the developer has typed past it, or nothing at all** — which turns out to rearrange every assumption a normal inference service is built on.

The design rests on one rule: **the unit of capacity is a warm session, not a request.** Inference is session-affine to the replica holding that editor's key-value prefix, and every other tier is stateless and disposable. That single boundary is why the inference tier is a placement problem rather than a balancing one, why scale-in is correctness-adjacent, why multi-region is a working-set decision rather than a latency one, and why the correct output under any failure is a fast, silent nothing.

The decisions that carry the design:

- **The warm session is the unit of capacity.** Between two keystrokes the prompt barely changes, so reusing the prefix is simultaneously the largest latency win and the largest cost win available. Requests are placed, not balanced (ADR-01).
- **The prefix cache is tiered and partitioned by tenant.** Device memory, host memory, node-local NVMe — and a hard tenant boundary, because a content-keyed shared cache would turn the fleet's chief efficiency mechanism into its chief isolation risk (ADR-02).
- **Prompt order is a contract, not a style.** The leading segment must be stable across keystrokes or the cache is invalidated on every one of them. Volatile context goes last, and the ordering rule is versioned and logged (ADR-03).
- **Refuse before generating; never return late.** Admission control compares the propagated deadline against queue depth and rejects at the door for zero accelerator cost, which makes a 4× burst a capacity question rather than a latency incident (ADR-04).
- **The untrusted client owns debounce and cancellation.** Only the editor knows the keystroke, so the timing mechanisms live there — and no policy, filter or quota is enforced only there (ADR-05).
- **Shed by refusing, in a declared order.** Multi-line before single-line, prefetch before user-initiated. Degradation is a published decision rather than a latency curve nobody chose (ADR-06).
- **Retention, not acceptance, is the quality metric.** A more confident model raises acceptance and can lower retention; acceptance alone cannot see the difference, which means it rewards plausibility — the exact failure mode of the technology (ADR-07).
- **Offline evaluation can only refuse.** Its correlation with retention is too weak to put a model in front of 1.2 million developers, and strong enough to catch a broken one for the price of a batch job (ADR-08).
- **A control arm is held back permanently.** Silent quality regression is the only failure class here with no error signal, so it needs a concurrent reference rather than a better alert (ADR-09).
- **Zero retention by absence, not by setting.** For an enterprise tenant the retaining path does not exist, including in logs, traces and crash dumps. A reviewer is right to discount a configuration claim, and this is not one (ADR-10).
- **Every filter declares its failure direction.** The secret scan is fail-closed inside 8 ms; the public-code filter is fail-closed for tenants that chose to block. An undeclared timeout is a policy chosen by whoever wrote the error handler (ADR-11).
- **Repository permission is resolved per request.** Against the requesting principal, at request time — not cached from index build. An index that outlives a permission change must not serve it (ADR-12).
- **Cost is measured per accepted suggestion.** Per-request cost hides the 55% that is discarded, which is where the money goes; wasted generation is a capped metric with an owner (ADR-13).
- **Two models, routed by request shape.** Deeper cascades pay twice inside a 300 ms budget, and a routing rule whose counterfactual never exists cannot be evaluated — so a forced-route sample manufactures one (ADR-14).
- **A region is a complete working set.** Prefix caches are never replicated: a region loss costs cold starts rather than correctness, and the fleet-fragmentation cost is the accepted price of the latency contract (ADR-15).

The architecture one-pager (including why the design should still hold up in ten years, and the five risks that would change it) and the full decision record appear on the landing page of the diagram set, directly below the index of views. The same content is published as [docs/architecture-one-pager.md](docs/architecture-one-pager.md) (~15 min) and [docs/decision-record.md](docs/decision-record.md) (~68 min).

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The landing page: 21 views in seven acts with every format linked, then the **architecture one-pager** and the **decision record** |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning cards, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same views as SVG with the diagram XML embedded; they re-open fully editable in diagrams.net |
| `diagrams/drawio/*.drawio` | draw.io source |
| `docs/architecture-one-pager.md` | The one-pager as markdown |
| `docs/decision-record.md` | The 15 decision records, the capability-to-technology table and the package glossary as markdown |
| `specs/part-a..c.json` | Diagram specifications, the source of truth for every view |
| `specs/manifest-a..b.json` | Acts, page titles, subtitles and reasoning cards |
| `specs/adr-onepager.json`, `specs/adr-records-a..b.json` | The one-pager, the decision records, the capability-to-technology table and the glossary |
| `scripts/build.sh` | Rebuilds every deliverable from the specs (Node only, no network) |
| `ask.md` | The requirement |

## The twenty-one views

| # | View | What it answers |
|---|---|---|
| 01 | System Context | Who types, what it depends on, and the three relationships it deliberately refuses — chat, billing and training |
| 02 | High-Level Architecture | Seven stages, with the admission gate deliberately sitting before inference rather than inside it |
| 03 | Actors and Their Journeys | Eight parties, two of them machines — including the promotion gate, which refuses a model without waking anyone |
| 04 | Journey — The Developer and the Grey Text | The trough is the wait, not the reading: a cold prefix reads as a broken editor rather than a slower model |
| 05 | Journey — Promoting a Model | The trough is Judge, where acceptance rises and retention falls and only one of them is telling the truth |
| 06 | Journey — Switching On a Tenant | The first trough is the security review, and it is answered structurally rather than with a setting |
| 07 | Layered Architecture | Eight layers, of which exactly one holds durable truth — and it is not on the request path |
| 08 | Platform Components | What is stateless, what is session-affine, and what is strongly consistent |
| 09 | Integration Surface | Four APIs, one of which is on a 300 ms budget and three of which may be slow |
| 10 | Data Flow — What Leaves the Machine | Three destinations: kept without code, kept with code by opt-in, and never kept |
| 11 | Data Architecture | Why the prefix caches are deliberately the one store with no recovery objective at all |
| 12 | Data Model | Twelve entities, none of which holds a line of source code |
| 13 | Critical Flow — One Keystroke | Nineteen messages, and the tenth is the refusal that keeps the latency contract |
| 14 | Context Assembly | Spending a token budget, and why the order of the spend matters more than the amount |
| 15 | Four Fates of One Keystroke | Five request populations over six stages; two cost nothing and one costs most of the fleet |
| 16 | Model Promotion and Gating | An offline gate that can only refuse, and an online verdict that arrives a day late |
| 17 | Deployment Topology | Three regions, each a complete working set rather than a fraction of one |
| 18 | Observability | Six signal classes against six stages, and the one row with no error signal behind it |
| 19 | Model and Quality Lifecycle | A loop that closes on retirement, because a mounted model holds working set the next candidate needs |
| 20 | Trust Zones and Network Segmentation | Five zones by decreasing exposure, and the plane that is allowed no egress at all |
| 21 | Identity, Seat and Policy Flow | Fifteen messages, and the permission check that is evaluated per request rather than per index build |

## Rebuilding

```bash
bash scripts/build.sh
```

Node 20+ and nothing else. No draw.io Desktop, no browser, no network. The build assembles
`specs/views.json` and `specs/manifest.json` from the authoring parts, generates the draw.io
sources, fails on any geometry or routing defect, renders the editable and plain SVGs, writes the
HTML pages and the index, injects the one-pager and the decision record, and proves every relative
link in `diagrams/` resolves.

At v1.0 the geometry gate reports 0 errors and 0 warnings across 21 files, the routing gate
0 errors and 0 warnings across 177 edges, all 389 nodes resolve to an icon with no weak matches,
and the link check passes on 166 relative links. Getting the routing gate to zero took several
rounds of cutting rather than tuning: an unlabelled intra-stage edge on the high-level view, a
duplicated return path, four relationship labels on the integration view moved into node subtitles,
and two zone-crossing edges on the segmentation view whose meaning the node subtitles and the view
note already carried.

## A note on the numbers

Every rate, latency, ratio, threshold and retention figure in this package is a **stated
assumption**, chosen to be defensible and arguable rather than measured. They are stated precisely
so that a reviewer can disagree with one and follow it to the decision that depends on it.
`ask.md` marks them as assumptions section by section; the decision record's evidence note
restates the operating context in one paragraph.