AI Code Completion Service

AI Code Completion Service · Solution Architecture v1.0 · Microsoft Azure · Developer Experience Architecture · 2026-10 · 21 views · 15 architecture decision records

The unit of capacity is a warm session, not a request: inference is session-affine to the replica holding that editor's KV prefix, and every other tier is stateless and disposable.

Everyone has met this system from the outside. You type three characters, grey text appears ahead of the cursor, and you either press Tab or keep typing and watch it vanish. It is the most-used AI product in the world by interaction count, and it is the only one that fires on a keystroke rather than on a prompt. That one fact rearranges every assumption a normal inference service is built on. The user's patience is not measured in seconds but in a couple of hundred milliseconds, because the alternative to a suggestion is not a worse suggestion — it is the developer having already typed the line themselves, at which point an arriving suggestion costs attention and returns nothing. More than half of all generated work is thrown away before anyone reads it, so the economics are set by cancelled requests rather than by completed ones. There are no labels: nobody tells the platform whether a suggestion was good, and the only evidence available is whether a developer took it and whether it survived the afternoon. And the input is the most sensitive artefact the customer owns, which means an enterprise's security reviewer holds a veto that no latency number will overturn. A design that treats this as "an LLM behind an API" gets all four of those wrong at once.

Make the warm session the unit the fleet is organised around, and make everything else abandonable. A request arrives at a global edge that spends sixty milliseconds on identity, seat, quota, client version and geography, then stamps a deadline that travels with it. Context is assembled against a declared token budget from the current file, open tabs, recent edits and permission-scoped repository retrieval, ordered so that the prompt's leading segment is stable across consecutive keystrokes. An affinity router asks which accelerator replica already holds that session's key-value prefix and places the request there; an admission gate refuses outright anything that cannot make its deadline, before a single accelerator second is spent. Generation streams back through trimming, suffix deduplication and a fail-closed secret scan, and lands as grey text. Everything durable sits beside that path and never on it: policy and seats in a strongly consistent store read through a thirty-second cache, models in an immutable content-addressed registry, and interaction telemetry in an append-only log whose code-bearing stream is separated at the boundary and, for enterprise tenants, does not exist. Under any failure the correct output is a fast, silent nothing.

What it is, and what it is not

• A placement problem over replicas holding warm prefixes — not A stateless inference pool behind a load balancer
• A service whose correct failure output is silence — not A service that returns an error the developer can see
• Work budgeted on the assumption that most of it is discarded — not Work priced per completed request
• Accepted-character retention measured against a held-back control arm — not An acceptance rate on a dashboard
• A zero-retention promise made by the absence of a path — not A retention setting switched off
• Repository permission resolved per request — not An index built once under the permissions of the day
• Filters that each declare whether they fail open or closed — not A filter chain that times out
• Cost expressed per accepted suggestion — not Cost expressed per request served
• Three regions, each a complete working set — not One global fleet with regional front doors

The decisions that are the architecture

1. The warm session is the unit of capacity (ADR-01) — Inference is session-affine to the replica holding that editor's prefix, which makes the inference tier a placement problem rather than a balancing one and makes scale-in correctness-adjacent. It is the one decision every later view either honours or contradicts.
2. The prefix cache is tiered and partitioned by tenant (ADR-02) — Device memory, then host memory, then node-local NVMe — and a hard tenant boundary, because a content-keyed shared cache would turn the fleet's chief efficiency mechanism into its chief isolation risk.
3. Prompt order is a contract, not a style (ADR-03) — The leading segment of the prompt must be stable across consecutive keystrokes or the cache is invalidated on every one of them. Volatile context goes last, and the ordering rule is versioned and logged.
4. Refuse before generating; never return late (ADR-04) — Admission control evaluates the remaining deadline against queue depth and rejects at the door for zero accelerator cost. A 4× burst becomes a capacity question instead of a latency incident.
5. The untrusted client owns debounce and cancellation (ADR-05) — Only the editor knows the keystroke, so suppression, debounce and cancel-on-keystroke live there — while no policy decision, filter or quota is enforced only there.
6. Shed by refusing, in a declared order (ADR-06) — Multi-line before single-line, speculative before user-initiated, lower tier before higher. Degradation is a published decision rather than a latency curve nobody chose.
7. Retention, not acceptance, is the quality metric (ADR-07) — Accepted characters still present after 120 seconds. A more confident model raises acceptance and can lower retention, and acceptance alone cannot see the difference.
8. Offline evaluation can only refuse (ADR-08) — The held-out suite is a promotion gate, never a verdict: its correlation with retention is too weak to put a model in front of 1.2 million developers.
9. A control arm is held back permanently (ADR-09) — Silent quality regression is the only failure class in this system with no error signal. The answer is a standing control arm and automated rollback, not an alert.
10. Zero retention by absence, not by setting (ADR-10) — For an enterprise tenant the retaining path does not exist — including in logs, traces and crash dumps. A reviewer is right to discount a configuration claim, and this is not one.
11. Every filter declares its failure direction (ADR-11) — The secret scan is fail-closed inside an 8 ms budget; the public-code filter is fail-closed for tenants that chose to block. Which way a filter fails is a product decision, never an implementation accident.
12. Repository permission is resolved per request (ADR-12) — Against the requesting principal, at request time — not cached from index build. An index that outlives a permission change must not serve it.
13. Cost is measured per accepted suggestion (ADR-13) — Per-request cost hides the 55% that is cancelled, which is where the money actually goes. Wasted generation is a capped, budgeted metric with an owner.
14. Two models, routed by request shape (ADR-14) — A fast single-line model and an escalated multi-line model. Deeper cascades pay twice at the tail and cannot be evaluated, because the counterfactual suggestion is never generated.
15. A region is a complete working set (ADR-15) — Prefix caches are never replicated across regions. Losing a region costs cold starts rather than correctness, and the fleet-fragmentation cost is accepted as the price of the latency contract.

Why this should still be right in ten years

The model, the serving runtime, the accelerator, the editor and the cloud will all be replaced inside this platform's life. These are the properties that should outlast them.

• Locality beats statelessness when the state is expensive to rebuild. "Route to where the warm state already is" is not a property of today's attention implementation. Any generative decoder will keep per-sequence state that is cheaper to reuse than to recompute, and any architecture serving an incremental, high-frequency editing signal will find that the previous request's state is almost the whole of the next one's. The specific mechanism — key-value cache, compiled state, a learned summary — will change. The decision that clients are placed rather than balanced survives all of them.
• A deadline that travels with the request outlives the service that honours it. Deadline propagation plus refuse-rather-than-queue is a protocol property, not a framework feature. It is what makes capacity planning honest: load shows up as refusals you can count rather than as a latency distribution that degrades silently. This pattern was right before this product existed and will be right after it.
• Behavioural measurement is the only thing that scales with model capability. Every offline benchmark eventually saturates or is trained against. What does not saturate is whether a human kept the output. Retention against a held-back control arm will still be the right instrument when the model writes whole functions, because the question it answers — did this survive contact with the work — does not depend on what the model can do.
• Privacy by absence survives audits that privacy by configuration does not. A control implemented as a missing code path cannot be misconfigured, cannot drift, and does not need to be re-attested after every refactor. As regulation around training data and source-code provenance tightens, the designs that will still be adoptable are the ones where the sensitive path was never built.
• Unit economics expressed in the user's unit, not the machine's. Cost per accepted suggestion will outlive the accelerator generation, the model size and the pricing model, because it is denominated in the thing the customer is buying. Cost per request, cost per token and cost per GPU-hour all stop being comparable the moment any of those three change.

Non-functional targets

Every figure below is a stated assumption. They are listed with the mechanism that is supposed to deliver them and the view where that mechanism is drawn, so a reviewer can disagree with a number and follow it straight to the design that depends on it.

Scope

In scope

• The editor-facing request lifecycle: a streaming, cancellable fill-in-the-middle suggestion API with client-supplied deadlines, and the published client contract (debounce, single in-flight per session, cancel-on-keystroke, suppression rules, version floor)
• Context assembly from the open workspace: current file, open tabs, recent edits, imported symbols and permission-scoped repository retrieval, under an enforced token budget with priority truncation and stable-prefix ordering
• Low-latency inference serving: continuous batching, a tiered key-value prefix cache, session-affine placement, deadline-aware admission control and an ordered shedding ladder
• Model portfolio and routing: an immutable content-addressed registry, request-shape routing between a fast and an escalated model, traffic splitting, canary gating and ten-minute rollback
• Suggestion safety: secret scanning, public-code match filtering with warn and block modes, suffix deduplication and degenerate-output detection, each with a declared failure direction
• Measurement: display/cancel/accept/retain telemetry, accepted-character retention, randomised online experiments with stable assignment, and a promotion gate with automated rollback
• Tenancy, privacy and administration: zero-retention mode, geography pinning, seats, policy with declared precedence, consent and audit, and per-tenant cost attribution

Explicitly out of scope

• Training, fine-tuning or adapting the models — there is deliberately no data path from tenant code to training
• The chat and agent surfaces that sit beside completion in the same editor: they share an editor and almost nothing else, with a latency budget measured in seconds
• The editor plugins themselves, beyond the contract they must honour and the version gate that enforces it
• Code review, static analysis and security scanning of the developer's own code
• Billing and invoicing: usage figures are exported nightly to a billing platform that owns the money
• Per-repository model adaptation, next-edit prediction and an on-device fallback model, all named in phase 3 of the requirement and deliberately deferred

What a four-week prototype should prove

The prototype's job is to falsify the three central claims — that session affinity is worth the operational cost it imposes, that refusing at the door really does hold the latency contract under burst, and that retention is measurable well enough to gate a rollout — on one language and a few hundred developers, not to build a platform.

1. One editor (VS Code), one language, one model, one region, 200 volunteer developers, with client-measured first-token latency as the headline number from day one
2. Session-affine routing against a measured baseline of stateless routing on the same hardware, reporting the p95 difference and the cost of a deliberate mid-session drain
3. A deadline-aware admission gate under a synthetic 4× burst, proving that refusals rise while p95 for admitted requests stays inside budget — and counting the accelerator seconds the refusals saved
4. Stable-prefix prompt ordering against a deliberately volatile ordering, to put a real number on the cache-invalidation cost the ordering rule exists to prevent
5. Accepted-character retention instrumented end to end, including the 120-second probe, with a held-back control arm large enough to detect a 1.5% relative acceptance change
6. A zero-retention tenant with an independent review of logs, traces and crash dumps for code-bearing content — the one test whose failure invalidates the adoption argument rather than the performance one

• A GPU replica holding warm sessions is drained under live traffic: every affected session continues on another replica with a measured latency penalty and zero errors, and the penalty is within the 180 ms cold-prefix budget rather than a full recompute
• A synthetic 4× burst is applied: admission refusals rise, p95 first-token latency for admitted requests does not move, and the accelerator seconds spent on refused requests is zero — which is the entire argument for refusing over queueing
• The prompt recipe is switched to a deliberately volatile ordering with everything else held constant: the prefix hit rate must collapse and first-token p95 must rise by roughly the cold-prefix penalty, confirming that prompt order is a latency decision and not a style one
• A developer's repository access is revoked while they are mid-session: the next keystroke's retrieval must no longer return content from that repository, without waiting for an index rebuild
• The secret scanner is taken offline: no suggestion is displayed to any tenant, the suppression is counted, and the outage appears as a product outage rather than as a quiet stream of unfiltered completions
• A zero-retention tenant is driven through a forced crash, a trace capture and a full debug-logging cycle: an independent reviewer searching every sink finds no code-bearing content, which is the test whose failure invalidates the adoption argument rather than the performance one

Open risks, carried rather than hidden

The reasoning behind every component and technology choice is in the Architecture Decision Record: 15 records across 5 areas, each with the alternatives that lost and what the choice costs.

AI Code Completion Service · Solution Architecture v1.0 · Microsoft Azure · Developer Experience Architecture · 2026-10 · 21 views · 15 architecture decision records

The argument these decisions serve is summarised in the Architecture One-Pager.

Fifteen decisions, grouped by the question they answer, each with the forcing question, the alternatives, what would flip the choice, and why it should outlast the technology it is realised on.

Status of this document. This is a design, not a report on a running system. Every rate, latency, ratio, threshold and retention figure in this package is a stated assumption, chosen to be defensible and arguable rather than measured. They are stated precisely so that a reviewer can disagree with one and follow it to the decision that depends on it. The requirement document marks them as assumptions section by section; the operating context assumed throughout is 1,200,000 daily active developers across 40 enterprise tenants and a self-serve population, 2,400,000,000 suggestion requests a day at 35,000 per second steady state and 90,000 per second at the working-day peak, of which at least 55% are cancelled before anything is displayed, served from roughly 4,000 accelerators across three regions inside a client-measured first-token budget of 300 milliseconds at p95.

How to read a record

• Question: The forcing question: why a decision was needed at all.
• Context: The requirement, the scale and the constraint that make it hard.
• Decision: What this architecture does, stated so it can be checked.
• How it is realised on Azure: The concrete mechanism: which service or package, configured how, in which subscription.
• Options weighed: Chosen, rejected, deferred, or right elsewhere, with the reason for each.
• Consequences: What the choice buys and what it costs, both kept visible.
• Choose differently when: The conditions that would flip the decision for your system.
• Why it holds up over time: What keeps the decision right as scale, staff and technology change.
• Lesson: The principle that transfers beyond this platform.

Decision map

The affinity boundary: The decision that defines the architecture, and the two that follow from it directly: where the expensive state lives, and what the prompt owes it.

• ADR-01 · The unit of capacity is a warm session; inference is session-affine to the replica holding its prefix
• ADR-02 · The prefix cache is tiered across device, host and local disk, and partitioned by tenant as a hard boundary
• ADR-03 · The prompt's leading segment is contractually stable; volatile context is placed last

The latency contract: Who owns the deadline, what is returned when it cannot be met, and where the work is refused rather than slowed.

• ADR-04 · Admission control refuses before generating; a late suggestion is never returned
• ADR-05 · Debounce, single-in-flight and cancellation live in the untrusted client, and enforce nothing
• ADR-06 · Load is shed by refusing in a declared order, never by extending latency

Measuring a product with no labels: What counts as better, who is allowed to say so, and how a regression with no error signal is caught at all.

• ADR-07 · Accepted-character retention is the canonical quality metric; acceptance rate is a fast companion
• ADR-08 · The offline evaluation suite is a promotion gate that can only refuse
• ADR-09 · A control arm is held back permanently, because silent quality regression has no error signal

What may be retained, and what may be suggested: The two promises an enterprise actually buys: that its code is not kept, and that its own secrets are not handed back to it.

• ADR-10 · Zero retention is implemented as the absence of a path, not as a configuration setting
• ADR-11 · Every filter declares its failure direction; the secret scan is fail-closed inside an 8 ms budget
• ADR-12 · Repository permission is resolved per request against the requesting principal, not cached from index build

Cost, capacity and the fleet: Where the money goes in a workload whose majority output is discarded, and how many fleets it takes to hold the latency contract.

• ADR-13 · Cost is measured per accepted suggestion, and wasted generation is a capped, budgeted metric
• ADR-14 · Two models routed by request shape — not one model, and not a deep cascade
• ADR-15 · Each region is a complete working set; prefix caches are never replicated across regions

Technology by capability

Microsoft Azure was chosen for this exercise for two reasons that point the same way. The first is rotation: across this repository's use cases Amazon Web Services and Google Cloud have dominated the last dozen packages, and Azure has been the least exercised of the three hyperscalers — reaching for the same catalogue repeatedly teaches that catalogue rather than architecture. The second is that this topic genuinely belongs here: the reference implementation of inline code completion at this scale runs on Azure GPU capacity, and the identity, seat and tenancy model that an enterprise rollout depends on is the one most enterprises already operate in Entra ID. The requirement document stays vendor-neutral throughout — "accelerator pool", not a VM SKU — and the table below is the one place where the architecture commits to products. Every row names what would be used instead elsewhere, because the design should survive a change of cloud and this is where that claim is tested.

The decisions, and the alternatives that lost

The affinity boundary

The decision that defines the architecture, and the two that follow from it directly: where the expensive state lives, and what the prompt owes it.

ADR-01 · The unit of capacity is a warm session; inference is session-affine to the replica holding its prefix

Status: Accepted · Shown on views: 02, 08, 15, 17

Between two keystrokes the prompt barely changes. Does the fleet exploit that, and if it does, what is given up to make the exploitation possible?

Context. A completion request's prompt is thousands of tokens of surrounding code. The next request in the same session, sixty milliseconds later, is the same thousands of tokens plus one character. Recomputing attention over the whole prefix on every keystroke is affordable at a demo's traffic and ruinous at 2.4 billion requests a day: it is simultaneously the largest latency cost and the largest money cost in the system. Reusing the cached prefix state is therefore not an optimisation but the economic basis of the product. The difficulty is that the cached state lives in a particular replica's memory, which means the request must go where the state is. That one requirement breaks the assumption every other tier in a modern platform is built on — that any replica can serve any request — and it does so in the tier that is hardest to operate: the one holding four thousand accelerators.

Decision. Inference is session-affine. A session identifier, stable for the editing session, is routed to the replica that holds that session's key-value prefix; capacity is planned, scaled and alerted on in units of concurrent warm sessions rather than requests per second. Every other tier — edge, context assembly, post-processing, telemetry ingest — is stateless and disposable. When no replica holds the session, any replica may take it and pay the cold-prefix cost; affinity is a preference with a measured penalty, never a queueing constraint.

How it is realised on Azure. AKS GPU node pools run the model servers, with the affinity map held in Azure Cache for Redis as (tenant, session) → replica with a session-length TTL. The affinity router runs in the stateless request plane on Container Apps and resolves placement before admission. Horizontal pod autoscaling is driven by custom metrics — admission rejection rate and queue-wait at p99 — published to Azure Monitor, not by GPU utilisation. Node pools use a drain protocol that stops accepting new sessions while completing in-flight sequences, with the eviction surfaced as an affinity-loss counter rather than as a pod restart.

What it buys

• An 85% prefix hit rate makes p50 first-token latency 120 ms rather than a full recompute, and takes the dominant term out of the cost model at the same time
• Capacity planning becomes honest: the question "how many developers can this fleet hold" has an answer in warm sessions, which is a quantity operators can reason about
• Cancellation becomes cheap. Evicting a sequence while keeping its prefix warm means the 55% of requests nobody reads cost a batch slot rather than a recompute

What it costs

• Scale-in is correctness-adjacent: removing a replica evicts live sessions, so autoscaling must be asymmetric and drains must be graceful rather than terminated
• The inference tier needs placement logic, an affinity store, and a failure mode — affinity loss — that no other tier has
• Multi-region stops being a latency decision and becomes a working-set fragmentation decision (ADR-15)

Choose differently when. If the prompt were not incremental — a product where each request's context is unrelated to the last, such as a batch code-explanation service or a one-shot refactoring tool — there would be no warm prefix to route to, and a stateless pool would be strictly better. The decision is justified by the keystroke cadence, and it should be revisited the moment the dominant surface stops being per-keystroke. It would also flip if an interconnect made transferring a prefix cheaper than recomputing it: at that point the shared remote cache becomes the better design, keeping the reuse and giving back painless drains.

Why it holds up over time. "Place the request where the expensive state already is" does not depend on today's attention implementation. Any generative decoder keeps per-sequence state, and any system serving an incremental high-frequency editing signal will find the previous request's state is almost all of the next one's. The mechanism may become a compiled state, a learned summary or something unnamed; the decision that clients are placed rather than balanced survives the change. What would not survive is making the cached state authoritative — the moment a prefix cannot be recomputed, affinity loss becomes an error instead of a latency penalty, and the whole design stops degrading gracefully.

Lesson. When the state a request needs is expensive to rebuild and cheap to keep, stop balancing and start placing — and keep the state recomputable, so that losing it costs latency rather than correctness.

ADR-02 · The prefix cache is tiered across device, host and local disk, and partitioned by tenant as a hard boundary

Status: Accepted · Shown on views: 07, 11, 15, 20

Device memory is the scarcest resource on the fleet and it has to hold both the batch and the cached prefixes. How is it divided, and what happens to a prefix that is evicted?

Context. A replica's accelerator memory is contended three ways: model weights, in-flight batch activations, and cached prefixes. Give more to prefixes and the batch narrows, which lowers throughput; give more to the batch and the hit rate falls, which raises latency. Meanwhile the obvious efficiency win is tempting and dangerous: prefixes are mostly code, much of it shared — the same framework imports, the same license header, the same generated client. A cache keyed purely on content would deduplicate across tenants and raise the hit rate materially. It would also mean one tenant's cached state can be reached through another tenant's request, which converts the fleet's chief efficiency mechanism into its chief isolation risk, in a product whose input is the customer's source code.

Decision. The prefix cache is three tiers — accelerator device memory, host DRAM, then node-local NVMe — with eviction cascading downward rather than discarding. The cache key includes the tenant identifier as a hard partition: no cache entry is ever reachable from a request belonging to another tenant, regardless of content equality. Cross-tenant deduplication is refused as a design, not configured off. Device-memory allocation between batch width and prefix residency is a tuned, published parameter with the hit rate and batch occupancy both reported.

How it is realised on Azure. Model servers on AKS GPU node pools manage the device tier in-process and spill to host DRAM, then to the node's local NVMe temp disk, which is sized for the working set rather than for logs. Tenant partitioning is a prefix on the cache key, enforced in the server rather than by convention. Tier hit rates (HBM / DRAM / NVMe) and batch occupancy are exported to Azure Monitor as first-class capacity signals, and the device-memory split is a per-model-pool configuration value in the registry's decoding config rather than a deployment flag.

What it buys

• Eviction degrades instead of resetting: a prefix pushed out of device memory is a disk read, not a recompute, which is what keeps the cold-prefix penalty at 180 ms rather than several hundred
• The tenant isolation claim is enforced by the key rather than by a policy, which is the form a security reviewer can actually accept
• The batch-versus-residency split becomes a visible, tunable number with two reported metrics, instead of an emergent property nobody owns

What it costs

• A measurable hit-rate gain is given up permanently, and with it some accelerator spend
• Node-local NVMe becomes a sizing requirement on the GPU node pool, which constrains instance selection
• Three tiers means three hit rates to reason about, and a regression in the lowest tier is easy to miss because it only shows up at the tail

Choose differently when. In a single-tenant deployment — a sovereign or on-premises installation for one organisation, which is phase 3 of the requirement — the partition buys nothing and content-addressed deduplication becomes straightforwardly correct. The decision is a multi-tenancy decision, not a caching one. It would also change if device memory stopped being the binding constraint: with an order of magnitude more of it, the tiering below device memory would matter much less and could be dropped for simplicity.

Why it holds up over time. "Efficiency mechanisms that cross a trust boundary are not efficiency mechanisms" is the durable half of this record. The specific tiers will be replaced — memory hierarchies change every accelerator generation, and the names HBM, DRAM and NVMe will not last — but the rule that a cache key carries the tenant, and that sharing is refused rather than configured, will still be the right call when the hierarchy has four levels and different names.

Lesson. Put the trust boundary in the cache key, not in the policy that guards the cache. A deduplication you refuse to build cannot be enabled by a mistake.

ADR-03 · The prompt's leading segment is contractually stable; volatile context is placed last

Status: Accepted · Shown on views: 07, 13, 14

Prompt composition is usually treated as a quality question. What happens to the cache when it is treated only as a quality question?

Context. Context assembly has freedom in what it includes and in what order. Treated as a relevance problem, the natural instinct is to put the most relevant material closest to the cursor — retrieved snippets, the most recently edited region, whatever the ranker scored highest. All of those change between keystrokes. A prefix cache is valid only up to the first token that differs, so a single volatile item placed early invalidates the entire cached prefix on every keystroke, and the 85% hit rate the whole architecture depends on collapses to near zero. The failure is invisible in testing: quality metrics look fine, latency looks fine at low traffic, and the cost and tail latency only appear at scale. This makes prompt ordering an architectural constraint masquerading as an authoring preference.

Decision. The prompt has a declared two-part structure: a stable leading segment — the file's identity, the language preamble, the imported-symbol block, the retrieved repository snippets selected for the session rather than for the keystroke — followed by a volatile tail holding the cursor neighbourhood. Sources are assigned a priority that determines both truncation order and position, and the lowest-priority source is dropped first with the drop recorded. The ordering rule is versioned as a prompt recipe, and the recipe version is logged against every request so a quality change is attributable to composition rather than to the model.

How it is realised on Azure. The prompt assembler runs in the stateless request plane on Container Apps; the recipe is versioned configuration in the Azure SQL control plane, read through the same bounded-staleness Redis cache as policy, so a recipe change takes effect within 30 seconds without a deployment. Retrieved snippets are selected once per session and cached with the session rather than re-ranked per keystroke. The recipe version is a column on suggestion_request in the telemetry model; prefix stability is exported as its own metric so that a recipe change that hurts the cache is visible within minutes rather than in next month's bill.

What it buys

• The hit rate the architecture depends on is protected by a rule rather than by hoping, and prefix stability has a metric that fails loudly
• A quality regression can be traced to a recipe version without retaining a single prompt, which keeps attribution compatible with zero retention
• Retrieval becomes a per-session cost rather than a per-keystroke one, which also removes it from the critical path at the frequency that matters

What it costs

• The ranker cannot act on the most recent keystroke, so some genuinely better context is excluded by position
• Per-session snippet selection goes stale within a long editing session, and refreshing it is a deliberate cache-invalidating event that has to be scheduled
• Prompt authoring becomes a constrained activity with a review step, which slows down the experiments that quality work depends on

Choose differently when. If prefix reuse were abandoned (see the flip on ADR-01), this constraint evaporates and prompt order becomes purely a relevance question. Short of that, it would loosen if models became robust to context position in a way that let the volatile material sit anywhere — but that is a claim about relevance, not about caching, and the cache would still prefer the stable ordering.

Why it holds up over time. The transferable property is that a cache's hit rate is a function of the request's construction, not of the cache's implementation — so whoever composes the request owns the hit rate whether they know it or not. That will be true of any prefix-reusing system. Making the composition a versioned, measured artefact rather than code is what keeps it true as the team that writes prompts changes.

Lesson. If a cache keys on a prefix, the order of what you put in the prefix is an architectural decision. Hand it to whoever owns the latency budget, not to whoever owns relevance.

The latency contract

Who owns the deadline, what is returned when it cannot be met, and where the work is refused rather than slowed.

ADR-04 · Admission control refuses before generating; a late suggestion is never returned

Status: Accepted · Shown on views: 02, 13, 15

When the fleet cannot serve a request inside its deadline, what does the developer get?

Context. The usual answer in a serving system is to queue: accept the work, do it as fast as possible, and let latency degrade under load. That answer is wrong here, because the value of a suggestion falls to zero and then goes negative. A suggestion that arrives after the developer has typed past it is not a slow success — it is grey text appearing over a line they have already written, which costs attention and offers nothing. Worse, under burst the queueing design spends accelerator seconds on suggestions nobody will read, so the fleet is most wasteful exactly when it is most contended. The 55% baseline cancellation rate means this is not an edge case: the system's normal state is that most work in flight is already doomed.

Decision. Every request carries the client's remaining deadline. An admission gate compares that deadline against current queue depth and estimated generation time and refuses immediately — before any accelerator work begins — anything it predicts will miss. A refused request returns an explicit empty result, which the client treats as "the model had nothing to say". Nothing is queued without bound, and no result is returned after its deadline has passed, even if it is ready.

How it is realised on Azure. The deadline is stamped at the Front Door edge after identity and quota, propagated as a request header through the Container Apps request plane, and evaluated by the admission gate immediately before placement. The gate's inputs are the replica's in-flight sequence count, measured queue-wait at p99 from Azure Monitor, and a per-model generation-rate estimate from the registry's decoding config. Refusals are a first-class metric and the primary autoscaling signal for the GPU node pools (ADR-01); the client translates an empty result into no ghost text and widens its own debounce on repeated refusals.

What it buys

• A 4× burst becomes a capacity question with a number attached — the refusal rate — rather than a latency incident with a shrug attached
• Accelerator seconds are not spent on suggestions that cannot be displayed, which is what makes the cost-per-accepted target reachable
• The refusal rate is a leading autoscaling and alerting signal, which average GPU utilisation can never be

What it costs

• Generation time must be estimated before generation, from features available at admission; a bad estimator either refuses work it could have served or admits work it cannot
• Developers in a contended region experience the product as intermittently absent, which is a silent degradation they cannot report usefully
• The empty result is indistinguishable from "the model genuinely had nothing", so the refusal rate has to be instrumented server-side or the degradation is invisible

Choose differently when. If the product's value did not decay inside a second — a batch completion pass over a file, a pull-request suggestion run, an agentic edit the developer waits for — queueing would be correct and refusing would be perverse. The decision belongs to the keystroke surface specifically. It would also soften if accelerator capacity became cheap enough that peak could simply be provisioned for, at which point the gate still exists but almost never fires.

Why it holds up over time. Deadline propagation plus refuse-rather-than-queue is a protocol property, not a framework feature, and it is right wherever the value of a response decays faster than the queue drains. It has outlasted several generations of serving infrastructure already. The part that would date is the estimator, which depends on the decoder; the part that will not is making the refusal countable, because that is what lets capacity be planned honestly.

Lesson. If a response loses its value with age, admission control is not back-pressure — it is the feature. Refuse early, count the refusals, and never deliver something that has already expired.

ADR-05 · Debounce, single-in-flight and cancellation live in the untrusted client, and enforce nothing

Status: Accepted · Shown on views: 01, 09, 13, 20

The editor is the one component the platform cannot control and the only one that knows the keystroke. How much of the latency contract is allowed to depend on it?

Context. Three of the mechanisms that make this system affordable can only exist in the editor. Debounce requires knowing that a keystroke happened and that no further keystroke followed for sixty milliseconds. Cancellation requires knowing that the developer kept typing. Suppression inside string literals, comments and already-complete lines requires the editor's own parse of the buffer. None of these can be done server-side at this budget, because the information arrives with the keystroke and the round trip is the thing being avoided. But the editor plugin is also the least trustworthy component in the system: it runs on the developer's machine, it ships on its own release cadence, and a single bad version can change fleet economics by tens of percent with nothing server-side having changed.

Decision. The client owns debounce, single-in-flight-per-session, cancel-on-keystroke and local suppression, and these are published as a versioned client contract with a supported version floor. No policy decision, filter, quota or authorisation is enforced only on the client: every one of them is independently enforced server-side, so a modified or defective client can waste capacity and cannot gain access or bypass a filter. Clients identifying a version below the floor are refused rather than served degraded, and a server-side kill switch can force any client version into a reduced mode without a client release.

How it is realised on Azure. The contract is versioned alongside the Suggestion API and enforced at the Front Door edge, which reads the client version header and consults the flag service before anything else. The version floor, the reduced-mode flag and the per-tenant and per-developer kill switches live in the Azure SQL control plane and reach the edge through the policy cache, taking effect for new requests within 30 seconds. Per-seat rate limiting at the edge bounds what a misbehaving client can cost; cancellation-rate distribution is reported per client version so a bad release is visible as a distribution shift rather than as an unexplained spend increase.

What it buys

• The timing mechanisms sit where the information is, which is what makes the 300 ms budget attainable at all
• Authorisation, policy and filtering remain enforceable against a client the platform does not control, so adoption does not rest on client integrity
• The version floor and reduced mode give an operator two blunt but immediate levers when a release goes wrong

What it costs

• Accelerator spend is partly determined by client behaviour the platform can only observe after the fact
• Version skew is permanent across four editors, so the contract must stay backward-compatible for longer than is comfortable
• Every policy must be implemented twice — once in the client for latency, once on the server for truth — and the two can drift

Choose differently when. If the platform shipped the editor itself, the client would stop being untrusted and much of the duplication could go. That is the situation for a vendor whose own IDE is the only surface, and their design would legitimately differ. It would also flip if a server round trip got cheap enough — on a local network, or for an in-browser IDE served from the same region — that debounce could move server-side.

Why it holds up over time. "Put the mechanism where the information is; put the enforcement where the trust is" is the durable statement, and it will survive every editor, protocol and plugin API in this space. What dates is the specific list of mechanisms: a future client may own more of the pipeline, or less, but the split between latency-driven placement and trust-driven enforcement stays.

Lesson. A component can be load-bearing and untrusted at the same time. Let it carry the timing it alone can see, and never let it carry a decision you would have to defend.

ADR-06 · Load is shed by refusing in a declared order, never by extending latency

Status: Accepted · Shown on views: 02, 15, 18

When there is not enough capacity for everything, who loses their suggestion first — and is that a decision or an accident?

Context. Admission control (ADR-04) decides whether a single request can make its deadline. It does not decide which requests should be sacrificed when the fleet as a whole is short. Without an explicit answer, the answer emerges from queue mechanics: whoever happens to arrive during the spike loses, which means the loss is distributed by luck and is impossible to explain to a tenant afterwards. Worse, the most expensive work — multi-line suggestions, speculative prefetch — survives at the same rate as the cheapest, so shedding recovers far less capacity than it costs in goodwill. A fleet serving 1.2 million developers with a 5× trough-to-peak daily swing will be short regularly enough that this needs to be a design rather than an emergent property.

Decision. Shedding is ordered and published: speculative prefetch is shed first, then multi-line suggestions, then lower service tiers, and single-line user-initiated requests on the highest tier are shed last. Shedding is always a refusal — an explicit empty result — and never an extension of latency. A tenant that has reached a spend ceiling degrades along the same declared ladder (smaller model, single-line only, then suspension) rather than being cut off without warning.

How it is realised on Azure. The shedding order is configuration in the Azure SQL control plane, read through the policy cache and evaluated by the admission gate alongside the deadline check. Request shape (single-line versus multi-line) is already known at admission because it determines model routing (ADR-14), so no extra classification step is needed. Shed counts are exported per class and per tenant to Azure Monitor, and the per-tenant degradation ladder is surfaced in the admin API so an administrator sees which rung they are on before they see the bill.

What it buys

• Degradation becomes a decision with an owner and a published order, which is the difference between a managed shortage and an incident
• Shedding the expensive classes first recovers more accelerator capacity per refused request, so the shortage ends sooner
• Spend-ceiling behaviour and capacity-shortage behaviour share one ladder, so there is one degradation story rather than two

What it costs

• Multi-line suggestions are structurally the least reliable feature of the product, which is a permanent asymmetry users will notice
• Request-class value is a judgement encoded in configuration, and a wrong ordering is only discoverable from acceptance data afterwards
• Tier-based shedding couples the architecture to the commercial packaging, which then becomes hard to change

Choose differently when. If capacity were elastic on the timescale of a burst — accelerators available in seconds rather than minutes — shedding would matter far less and provisioning would do the work instead. It would also change if multi-line suggestions turned out to carry most of the measured retention, in which case the ordering is simply wrong and the data should overturn it.

Why it holds up over time. "Shed by value, in an order you published before the incident" outlives every scheduler. The specific classes here are tied to this product's shapes, but the principle — that a shortage should be resolved by a decision made in advance rather than by queue mechanics in the moment — is one of the few operational rules that has never stopped being true.

Lesson. Decide who loses before you are short. An unplanned shortage is distributed by luck, and luck is the one allocation policy nobody can defend afterwards.

Measuring a product with no labels

What counts as better, who is allowed to say so, and how a regression with no error signal is caught at all.

ADR-07 · Accepted-character retention is the canonical quality metric; acceptance rate is a fast companion

Status: Accepted · Shown on views: 04, 16, 18, 19

Nobody tells this system whether a suggestion was good. What stands in for that, and what does the substitute get wrong?

Context. The available signals are all behavioural. A suggestion is displayed, or cancelled before display; if displayed it is accepted or ignored; if accepted it survives or is deleted. Acceptance rate is the obvious metric: it is immediate, abundant and easy to compute. It is also systematically misleading in the one direction that matters. A model that is more confident and more fluent produces suggestions that look right and get taken, and some fraction of those are deleted twenty minutes later when the developer discovers the API does not exist. Optimising acceptance therefore optimises plausibility, which is precisely the failure mode of the underlying technology. Retention — how much of an accepted suggestion still exists in the file after a delay — sees the difference, and arrives a day late.

Decision. Accepted-character retention is the canonical quality metric, measured at 120 seconds and again at 24 hours after acceptance. Acceptance rate and time-to-first-token are fast guard rails that may halt a rollout on their own but may never promote one. A model that improves acceptance and regresses retention is refused. Both metrics are always reported against a control arm (ADR-09) rather than against the model's own history.

How it is realised on Azure. The editor plugin probes the accepted span at 120 seconds and reports surviving characters as an asynchronous telemetry event to Event Hubs; the 24-hour measurement is derived from a later session's buffer state for the same span. Events land in Azure Data Explorer, where retention and acceptance are materialised per model digest, language, prompt recipe version and experiment arm. The promotion gate reads those aggregates; thresholds live in the control plane so a gate can be tightened without a deployment. Only character counts and span offsets are transmitted — never the characters themselves — which is what keeps the metric compatible with zero retention.

What it buys

• The metric that gates promotion measures whether the suggestion was useful rather than whether it was convincing, which is the only defence against optimising for plausibility
• Retention is computable from character counts and offsets alone, so quality measurement survives a zero-retention tenant
• Per-language and per-recipe retention makes it possible to see a model that improved the mean and ruined one language

What it costs

• A one-day decision latency, which bounds how fast models can be iterated and forces a capped canary in the meantime
• The probe depends on the client, so a plugin release can move the headline quality metric without the model changing
• A span that was accepted and then legitimately refactored counts as lost retention, which puts a noise floor under the metric

Choose differently when. If the product moved from inline completion to whole-function or agentic edits, the unit of measurement changes — a diff accepted and then reworked is normal rather than a failure — and retention at the character level stops being meaningful. At that point test outcomes and review signals become the right instrument despite their latency.

Why it holds up over time. Every offline benchmark eventually saturates or gets trained against. What does not saturate is whether a human kept the output. The question retention asks — did this survive contact with the work — is independent of what the model can do, so it will still be the right instrument when the model writes whole modules. The 120-second probe is an implementation detail; the choice of a survival metric over an uptake metric is not.

Lesson. When you cannot measure correctness, measure survival. Uptake tells you the output was persuasive; survival tells you it was right, and only one of those is worth optimising.

ADR-08 · The offline evaluation suite is a promotion gate that can only refuse

Status: Accepted · Shown on views: 05, 16, 19

An offline suite over held-out completion contexts is cheap and fast. Why is it not allowed to decide anything?

Context. Offline evaluation is attractive because it is fast, repeatable and available before any developer is exposed. The problem is that its correlation with retention is weak and unstable: exact-match and edit-distance scores reward a model that reproduces the held-out continuation, which is not the same as producing something the developer keeps, and the correlation shifts whenever the corpus, the languages or the model family changes. Treating such a score as a verdict means putting a model in front of 1.2 million developers on the strength of a proxy whose relationship to the real metric nobody can state. Treating it as nothing, however, wastes the one check that can catch an obviously broken candidate for a few dollars.

Decision. The offline suite is a gate with one direction of authority: it can refuse a candidate before it reaches a canary, and it can never promote one. A model that has not been evaluated cannot be promoted at all. Promotion requires an online retention verdict (ADR-07) measured against the control arm. Offline thresholds are set to catch gross failures — wrong language, degenerate output, collapse on a common construct — rather than to rank near-equivalent candidates.

How it is realised on Azure. The harness runs as a job against a held-out corpus built only from licensed material, stored in Blob Storage and versioned alongside the registry. A model version in Azure Container Registry carries an eval_run_id, and the Model API refuses a traffic split for any digest without one. Thresholds are control-plane configuration, so tightening the gate is a configuration change and not a pipeline release. Results are retained for two years so that a later retention surprise can be checked against what the offline suite said at the time — which is how the correlation gets re-estimated rather than assumed.

What it buys

• An obviously broken candidate is stopped for the price of a batch job, before any developer is exposed
• Nothing is promoted on a metric whose relationship to usefulness cannot be stated, which is the whole point of the asymmetry
• Keeping offline results alongside later retention outcomes turns the correlation into something measured rather than assumed

What it costs

• Every candidate pays a canary day, even one that is obviously better offline
• Near-equivalent candidates cannot be ranked before exposure, so canary capacity rather than compute becomes the limit on iteration speed
• A licensed-only corpus is narrower than production traffic, so the gate is blind to constructs that only appear in customers' code

Choose differently when. If an offline metric were ever shown to predict retention reliably across languages and model families, the asymmetry could be relaxed and promotion could move offline — which would transform iteration speed. That is a measurement result, and the retained eval history exists precisely so it could one day be demonstrated rather than claimed.

Why it holds up over time. The durable statement is about the direction of authority, not about the metric: a cheap proxy may veto and may not decide. That rule holds for every proxy in every system, and it is what prevents proxy metrics from quietly becoming objectives. It will survive the replacement of exact-match scoring by whatever succeeds it.

Lesson. Give a cheap proxy the power to refuse and never the power to approve. The asymmetry is what stops a proxy becoming the objective.

ADR-09 · A control arm is held back permanently, because silent quality regression has no error signal

Status: Accepted · Shown on views: 16, 18, 19

Which failure in this system produces no error, no alert and no page — and what instrument detects it?

Context. Every other failure class here announces itself. Latency misses show up in a percentile, saturation in admission refusals, dependency failure in a degradation counter, filter failure in a suppression count. Quality is different: a model that has become less useful looks exactly like a healthy fleet. Latency is fine, errors are zero, acceptance drifts by a percent and nobody notices for a month, by which time it is attributed to a tooling change or a seasonal shift. Comparing against the model's own history does not help, because the population drifts too — new tenants, new languages, a change in the mix of work. The only reliable reference is a concurrent population served by a known-good configuration.

Decision. A control arm is held back permanently, not per experiment: a stable, randomly assigned fraction of developers continue on the previous known-good model and prompt recipe. All quality figures are reported as differences against that arm. The gate that halts a rollout reads the difference, not the absolute. Measurement failures — a telemetry gap, a broken probe, an arm whose assignment has drifted — are alerted separately from quality regressions, so an on-call engineer can tell a broken instrument from a worse model.

How it is realised on Azure. Arm assignment is stable per developer and held in the Azure SQL control plane as experiment_arm rows with an is_control flag, so an assignment survives restarts, plugin upgrades and cache flushes. Aggregates in Azure Data Explorer are computed per arm and per language. The control arm runs on its own share of accelerator capacity with the previous model mounted, which is capacity permanently unavailable to the current one. Arm-balance and telemetry-coverage checks run continuously and page a different rotation from quality alerts.

What it buys

• The one failure class with no error signal acquires an instrument, and an automated rollback can act on it without a human noticing first
• Population drift stops being confounded with model quality, which makes a 1.5% effect claim defensible
• Separating measurement-failure alerts from quality alerts removes the worst minute in the system: not knowing whether the fleet or the thermometer is broken

What it costs

• A permanent slice of developers gets a deliberately older model, which is a real and ongoing product cost
• Control-arm capacity holds the previous model warm forever, taking working set from the current one (ADR-15)
• Permanent assignment creates a long-lived population that must be rebalanced carefully, because a drifted arm silently invalidates every figure computed against it

Choose differently when. If quality could be measured absolutely — a reliable automated correctness check on each suggestion — the control arm's main job would disappear and it could shrink to an availability probe. Short of that, the arm shrinks only if the developer population stops drifting, which for a growing product it will not.

Why it holds up over time. "A metric with no error signal needs a concurrent reference, not a historical one" is the transferable half, and it applies far beyond this product — to recommendation quality, ranking relevance and anything else judged by behaviour. The cost of the arm will fall as capacity gets cheaper; the need for it will not change at all.

Lesson. For any failure that produces no error, buy a concurrent control rather than a better alert. You cannot alert on a number you have nothing to compare against.

What may be retained, and what may be suggested

The two promises an enterprise actually buys: that its code is not kept, and that its own secrets are not handed back to it.

ADR-10 · Zero retention is implemented as the absence of a path, not as a configuration setting

Status: Accepted · Shown on views: 01, 10, 11, 20

An enterprise security reviewer is told retention is disabled. Why should they believe it, and what would make the claim checkable?

Context. The input to this system is the customer's source code, which is usually their most valuable asset and often the one with the strictest contractual controls. A reviewer evaluating adoption knows that a setting can be misconfigured, re-enabled by a deployment, bypassed by a new code path, or quietly contradicted by infrastructure nobody considered — crash dumps, request logs, distributed traces, a debugging flag left on in one region. They are right to discount a configuration claim, because the history of such claims is bad. Meanwhile the platform genuinely needs telemetry: without interaction data there is no quality measurement at all (ADR-07, ADR-09), and without quality measurement the product cannot improve.

Decision. Telemetry is split at the boundary into a code-bearing stream and a non-code stream, and for a zero-retention tenant the code-bearing path does not exist: no sink is configured, no buffer is allocated, and the scrub is enforced at the sink as well as at the source so that infrastructure which does not know about retention mode cannot create one. Zero retention is the default for enterprise tenants. Diagnostic retention is opt-in, bounded at 24 hours, and drawn in red on the data-flow view because it is the only route by which tenant code reaches durable storage. There is no data path at all from tenant traffic to model training, and evaluation corpora are built from licensed material only.

How it is realised on Azure. The post-processing tier emits two Event Hubs streams; the code-bearing stream has no configured sink for a zero-retention tenant, so the emit is a no-op rather than a filtered write. A scrubber enforces the same rule at the Data Explorer and Blob sinks, which is where crash dumps and trace exporters would otherwise land. Retention mode is a column on the tenant row in Azure SQL, read through the policy cache, and failing closed to zero retention when the cache is cold. Consent and configuration changes are written to an append-only audit store kept five years, so a reviewer can see when the mode changed and who changed it.

What it buys

• The privacy claim becomes structural rather than procedural, which is the form that survives an audit and a refactor
• Quality measurement still works, because retention and acceptance need character counts and offsets rather than characters
• A cold policy cache fails to zero retention, so an availability problem can never become a disclosure

What it costs

• A tenant-specific quality complaint cannot be investigated from retained examples, only from aggregates and a reproduction
• Evaluation corpora stay narrower than production traffic, so the offline gate is blind to constructs that only appear in customer code (ADR-08)
• Two telemetry streams and two enforcement points is more machinery than one flag, and the sink-side scrub must be re-verified whenever infrastructure changes

Choose differently when. A tenant that actively wants retention — to debug its own usage, or to build an internal corpus under its own legal basis — can opt in, and that is the diagnostic path. For a single-tenant sovereign deployment the whole question changes shape, because the retaining storage is the customer's own. The decision is about a shared multi-tenant platform holding other people's source code.

Why it holds up over time. A control implemented as a missing code path cannot be misconfigured, cannot drift, and does not need re-attesting after every refactor. As regulation around training data and code provenance tightens, the designs still adoptable will be the ones where the sensitive path was never built. That argument gets stronger with time, not weaker.

Lesson. The strongest version of a privacy guarantee is an absent capability. If a path exists, you are promising about a configuration; if it does not, you are describing the system.

ADR-11 · Every filter declares its failure direction; the secret scan is fail-closed inside an 8 ms budget

Status: Accepted · Shown on views: 02, 08, 18

A filter on a 300 ms critical path will eventually be slow or unavailable. What happens to the suggestion then, and who decided?

Context. Two filters sit between generation and display. A secret scan catches keys, tokens and private-key blocks that the model may reproduce from its context or its training — the failure where a developer is handed back a credential, sometimes one of their own. A public-code match filter suppresses suggestions that reproduce a substantial verbatim span of known public source, which some enterprises treat as a licence exposure and others ignore. Both run inside a budget measured in single-digit milliseconds. In most systems what happens when such a filter is slow or down is never decided: it times out, and whichever way the timeout happens to be coded becomes the policy. For one of these filters failing open is a security incident and for the other it is a commercial preference, so a single timeout behaviour cannot be right for both.

Decision. Each filter declares its failure direction explicitly as part of its configuration, and the choice is a product decision rather than an implementation detail. The secret scan is fail-closed: if it cannot run, no suggestion is displayed. The public-code match filter is fail-closed for tenants that selected block mode and fail-open for tenants in warn or off mode. Every suppression is counted and attributed, so suppression is visible rather than silent, and a filter that starts suppressing a large fraction of candidates is an alert rather than a mystery.

How it is realised on Azure. Filters run in the stateless post-processing tier on Container Apps, in-process rather than as a network hop, so the budget is reachable. Failure direction, mode and thresholds are per-tenant configuration in Azure SQL read through the policy cache, defaulting to the stricter setting when the cache is cold. Filter time is reported separately from inference time in Azure Monitor. The public-code match index is refreshed in batch from a licensed corpus in Blob Storage and is never on the critical path for a tenant in off mode. Suppression counts by filter, tenant and reason go to Data Explorer alongside the interaction telemetry.

What it buys

• The one failure that cannot be allowed — handing back a credential — is structurally prevented rather than left to a timeout's default
• Tenants with different licence postures get different behaviour without different code paths in the filter itself
• Counted suppressions turn a silent quality cost into a measurable one, which is what makes the false-positive budget enforceable

What it costs

• A secret-scan outage is a full product outage for every tenant, which makes that component as availability-critical as inference
• Filtering spends part of a very small budget, and the spend is on the path of every displayed suggestion
• Each filter needs two tested code paths and a per-tenant configuration, which is a permanent increase in surface

Choose differently when. If generation-time prevention matured enough to make verbatim reproduction of secrets and licensed spans structurally impossible, the post-hoc filters could shrink to an audit sample and the fail-closed coupling would disappear. For a tenant with no licence concern and no secrets in its repositories — rare, and usually wrong about both — the filters could run in sample mode.

Why it holds up over time. "Every control on a critical path declares which way it fails, and the declaration is a product decision" is a rule that outlives any particular filter. It is also one of the few security properties that gets cheaper to keep as a system grows, because it is enforced by configuration shape rather than by vigilance.

Lesson. Decide a control's failure direction when you build it, not when it fails. An undeclared timeout is a policy chosen by whoever wrote the error handler.

ADR-12 · Repository permission is resolved per request against the requesting principal, not cached from index build

Status: Accepted · Shown on views: 14, 20, 21

A retrieval index is built once and read many times. Whose permissions does it serve — the builder's, or the reader's at the moment they read?

Context. Retrieval over the wider repository is what lifts a completion from the current file to the codebase, and it is the feature most likely to cause a disclosure. An index built under one set of permissions outlives those permissions: a contractor leaves a project, a repository is made private, a monorepo path moves behind an access list. If permission is captured at build time, the index keeps serving the old answer until it is rebuilt, and rebuilds take up to an hour for the largest workspace. The alternative is to resolve permission per request against the system that actually owns it, which puts a synchronous call to a third party inside a 300 ms budget — on a path where the obvious optimisation, caching the answer, is also the exact mistake being avoided.

Decision. Repository permission is evaluated at request time against the requesting principal, using the source hosting system as the authority. The default retrieval scope is the workspace, not the tenant. When the permission authority cannot answer inside the assembly budget, retrieval degrades to open-tabs-only context and the degradation is counted as a quality signal rather than treated as a silent fallback. Index contents are never served on the basis of permissions recorded at build time.

How it is realised on Azure. The retrieval service in the request plane calls source hosting with the principal from the validated Entra token and receives a scoped repository list, then retrieves only within that scope from the per-workspace indexes. Short-lived negative results may be cached; positive grants are not cached beyond the request. Private endpoints carry the call, and its latency is budgeted inside the server-side 15 ms assembly allowance with the degradation path taking over beyond it. Degradation-to-open-tabs is exported per tenant to Azure Monitor and correlated with acceptance in Data Explorer, so the cost of the degradation is visible in quality rather than only in latency.

What it buys

• A revocation takes effect on the next keystroke rather than on the next rebuild, which is the only version of this control that is defensible in a review
• Indexes become permission-free artefacts, so they can be rebuilt, moved and shared across a workspace's developers without re-deriving an access list
• The degradation path is counted, so the cost of a slow authority appears as a quality number rather than as an unexplained acceptance dip

What it costs

• A third-party call sits inside the assembly budget, and its tail latency becomes this platform's tail latency
• The degradation to open-tabs context is invisible to the developer and visibly worse in acceptance, so a slow authority quietly degrades the product
• Negative-result caching is the only caching allowed, which forgoes most of the available saving

Choose differently when. If source hosting offered a reliable permission-change event, cached grants with push invalidation would be strictly better: the same correctness with none of the critical-path cost. For a single-tenant deployment where every developer can read every repository, the check is vacuous and should be skipped rather than optimised.

Why it holds up over time. "Authorise against the authority at the moment of access" is as old as access control and keeps being re-learned by every system that builds a derived index over protected data. Search platforms, data catalogues, vector stores and retrieval layers all face it, and the ones that cached the grant all eventually disclosed something. That will not change.

Lesson. A derived index inherits its source's data and none of its permissions. Resolve access against the authority, every time, or accept that the index will eventually serve something it should not.

Cost, capacity and the fleet

Where the money goes in a workload whose majority output is discarded, and how many fleets it takes to hold the latency contract.

ADR-13 · Cost is measured per accepted suggestion, and wasted generation is a capped, budgeted metric

Status: Accepted · Shown on views: 15, 18

More than half of what this fleet generates is never read. Which denominator makes that visible rather than invisible?

Context. The natural unit for an inference service is cost per request, or cost per thousand tokens. Both are actively misleading here. At a 55% cancellation rate, cost per request averages the cheap discarded work with the expensive useful work and produces a number that improves when the fleet generates more rubbish faster. Cost per token is worse, because it rewards verbosity. Neither denominator lets anyone answer the question an engineering leader and a tenant administrator both actually ask: what did the spend buy. And because cancellation is driven by client behaviour the platform does not control (ADR-05), a denominator that hides cancellation also hides the single largest lever on fleet cost.

Decision. The headline cost metric is cost per accepted suggestion, with cost per displayed suggestion reported alongside it. Wasted generation — accelerator seconds spent on suggestions that were cancelled or never displayed — is a named, capped, budgeted metric with an owner and an alert, targeted at no more than 18% of accelerator seconds. Cost is attributed to tenant, language and model so that model routing (ADR-14) can be argued on cost per accepted rather than on cost per token. Per-tenant spend ceilings degrade along the published ladder (ADR-06) rather than cutting off.

How it is realised on Azure. Accelerator seconds are attributed at the batch scheduler per sequence and tagged with tenant, model digest, language and request shape, then joined in Azure Data Explorer against the interaction outcomes that say whether the suggestion was displayed and accepted. The same figures feed the admin API and the nightly export to the billing platform, so the number an administrator reads is the number they are charged on. Eager cancellation reclamation (50 ms at p99) is the primary control on wasted generation; admission refusal (ADR-04) is the second, since a refused request costs nothing at all.

What it buys

• The dominant waste term becomes a named metric with a cap and an owner instead of an emergent property of the cancellation rate
• Model choice can be argued on cost per accepted suggestion, which is what makes "use the smaller model" a measurable claim rather than a preference
• Administrators are shown and billed on the same figures, which removes the most common source of distrust in a usage-based product

What it costs

• The metric requires joining cost telemetry to outcome telemetry, so a telemetry gap becomes a billing and reporting gap
• Acceptance rate sits in the denominator, so a quality regression shows up as a cost regression and the two have to be disentangled before either is acted on
• A tenant whose developers are simply fast typists looks expensive per accepted suggestion through no fault of the platform

Choose differently when. If cancellation fell to near zero — a surface where suggestions are explicitly requested rather than fired on a keystroke — cost per request becomes honest again and the extra machinery is unnecessary. The decision is forced by the cancellation rate, so it should be revisited if the dominant surface changes.

Why it holds up over time. Cost per accepted suggestion will outlive the accelerator generation, the model size and the pricing model, because it is denominated in the thing the customer is buying. Cost per request, per token and per GPU-hour all stop being comparable the moment any of those three change, which they do every year.

Lesson. Choose the denominator the customer would choose. A unit cost that improves when you waste more work faster is not a cost metric, it is a vanity metric with a currency sign.

ADR-14 · Two models routed by request shape — not one model, and not a deep cascade

Status: Accepted · Shown on views: 08, 15, 16

Single-line completions are the overwhelming majority and the cheapest; multi-line are rarer, more valuable and much more expensive. How many models serve that distribution?

Context. One mid-size model is operationally simple and uniformly priced, and it overpays for the common case while underperforming on the rare one. A deep cascade — try the smallest, escalate on a confidence signal — is cheapest at the median and pays twice at the tail, because an escalated request has already spent the small model's latency before the large one starts, inside a budget that has no room for it. There is also a measurement problem that is specific to this product: the counterfactual suggestion is never generated, so a wrong routing decision produces no evidence. If the fast model was routed a request the large model would have completed better, nothing in the telemetry says so — the developer simply did not accept, which looks identical to the model having nothing useful to offer.

Decision. Two models, routed once at admission on features available before generation: request shape (single-line versus multi-line intent), language, file size and tenant tier. No mid-generation escalation, because escalation spends the budget twice. Routing is configuration rather than code, and routing quality is evaluated by holding a sample of traffic on forced routes so that the counterfactual exists for the sample even though it never exists for an individual request.

How it is realised on Azure. A fast single-line server and an escalated multi-line server run as separate AKS GPU node pools, each with its own affinity map and prefix cache, so a session's warm state belongs to one pool. The routing rule is versioned configuration in Azure SQL read through the policy cache, evaluated by the affinity router before admission so that placement and model choice are one decision. A small percentage of traffic is assigned to forced-route arms through the same experiment machinery as model promotion (ADR-09), which is what makes routing quality measurable at all.

What it buys

• The majority case is served by the cheap model, which is the single largest structural saving available after prefix reuse
• Routing and placement are one decision, so a request is never routed to a model whose pool does not hold its warm prefix
• Forced-route sampling creates the only evidence that can show the routing rule is wrong

What it costs

• Two GPU pools means two working sets, two rollout surfaces and two sets of capacity decisions (ADR-15)
• A session that changes shape mid-edit may move pools and pay a cold prefix, so the routing feature set has to be stable within a session
• Routing quality is only measurable on the forced-route sample, so the rule is always evaluated on a small fraction of the evidence

Choose differently when. If one model could serve both shapes at the fast model's cost, two pools stop earning their complexity and should be collapsed immediately — this is a decision that should be re-tested at every model generation. Conversely, if a genuinely cheap confidence signal emerged before generation, a three-tier portfolio could beat two; the barrier is that the signal must be available at admission, not mid-stream.

Why it holds up over time. The durable part is the refusal to make a routing decision that cannot be evaluated: building the forced-route sample is what keeps the rule honest as the model portfolio changes. Routing by request shape rather than by mid-flight escalation is also a property of the latency budget rather than of the models, so it holds as long as the budget does.

Lesson. Do not build a routing rule you cannot evaluate. If the counterfactual never exists, manufacture it for a sample — otherwise the rule becomes folklore that nobody can disprove.

ADR-15 · Each region is a complete working set; prefix caches are never replicated across regions

Status: Accepted · Shown on views: 11, 17, 20

Cross-continental round-trip time alone can consume most of a 300 ms first-token budget. How many fleets does that imply, and what is the cost of the answer?

Context. A developer in Singapore served from Western Europe spends 150 to 200 milliseconds of the budget on network before a token is generated, which leaves nothing for assembly, admission and decoding. The latency contract therefore requires inference near the developer. But the accelerator fleet is the expensive, lumpy resource: splitting four thousand accelerators across three regions means three smaller fleets that batch less efficiently, three working sets rather than one, and three rollout surfaces for every model change. It also interacts badly with session affinity (ADR-01), because the warm-prefix working set is exactly the thing that fragments. Meanwhile geography pinning for data residency (ADR-10) forces some of the split regardless of latency, which means the question is not whether to split but whether to pretend the split is only about latency.

Decision. Each serving region is a complete working set: its own GPU pools, its own prefix caches, its own policy read replica and its own measurement path. Prefix caches are never replicated or transferred across regions — they are recomputable, so replicating them would couple regions for no durability gain. A region loss costs cold starts rather than correctness: sessions re-establish elsewhere with cold caches, inside a 5-minute RTO. The control plane has one write region with regional read replicas; the model registry is geo-replicated because weights are large, immutable and read everywhere.

How it is realised on Azure. Azure Front Door routes by geography and by tenant residency pinning rather than purely by latency, refusing rather than spilling when a tenant's declared geography is unavailable. Each region runs its own AKS GPU node pools, Container Apps request plane, and Redis affinity and policy caches; Azure Container Registry is geo-replicated so weights are pulled locally by digest. Azure SQL has one write region with read replicas per serving region, and Event Hubs plus Data Explorer are regional with aggregates consolidated asynchronously. Measurement is pinned to a separate zone within each region so a telemetry incident and a serving incident are not the same incident.

What it buys

• The latency contract is attainable for developers on three continents rather than one, which is the difference between a product and a demo
• A region loss is a cold-start event with a 5-minute RTO, because nothing in the region held authority over anything
• Data residency and latency are served by the same topology, so pinning is not an exception to the architecture

What it costs

• Three smaller fleets batch less efficiently than one large one, which is a permanent utilisation cost
• The warm-prefix working set fragments, so each region needs its own headroom and the sum exceeds what one fleet would need
• Every model promotion, rollback and configuration change has three surfaces, which multiplies the rollout machinery and the ways a ramp can be inconsistent

Choose differently when. If network latency to a central region fell below roughly 30 milliseconds for the served population — a single-country deployment, or a future where that is simply true — one fleet becomes better on every axis and the regions should be collapsed. The sovereign single-tenant deployment in phase 3 is the opposite case: a region of one tenant, where fragmentation is the whole point and efficiency is explicitly not the objective.

Why it holds up over time. "Replicate authority, recompute derived state" is the transferable rule, and it is what keeps a multi-region design simple: the only things that cross a region boundary are the small immutable ones. That will hold through any change in accelerator, model or cloud. What will change is the number of regions, which is a function of network physics and of where the customers are.

Lesson. Replicate what is authoritative and recompute what is derived. A cache that crosses a region boundary has bought you a coupling and sold you nothing.

Every package used, in one table

These terms are used precisely in this package. Several are used loosely in the wider literature on inference serving, and the difference matters when reading the decision records.