# Architecture Decision Record

*AI Code Completion Service · Solution Architecture v1.0 · Microsoft Azure · Developer Experience Architecture · 2026-10 · 21 views · 15 architecture decision records*

The argument these decisions serve is summarised in the [Architecture One-Pager](architecture-one-pager).

Fifteen decisions, grouped by the question they answer, each with the forcing question, the alternatives, what would flip the choice, and why it should outlast the technology it is realised on.

> **Status of this document.** This is a design, not a report on a running system. Every rate, latency, ratio, threshold and retention figure in this package is a stated assumption, chosen to be defensible and arguable rather than measured. They are stated precisely so that a reviewer can disagree with one and follow it to the decision that depends on it. The requirement document marks them as assumptions section by section; the operating context assumed throughout is 1,200,000 daily active developers across 40 enterprise tenants and a self-serve population, 2,400,000,000 suggestion requests a day at 35,000 per second steady state and 90,000 per second at the working-day peak, of which at least 55% are cancelled before anything is displayed, served from roughly 4,000 accelerators across three regions inside a client-measured first-token budget of 300 milliseconds at p95.

## How to read a record

- **Question:** The forcing question: why a decision was needed at all.
- **Context:** The requirement, the scale and the constraint that make it hard.
- **Decision:** What this architecture does, stated so it can be checked.
- **How it is realised on Azure:** The concrete mechanism: which service or package, configured how, in which subscription.
- **Options weighed:** Chosen, rejected, deferred, or right elsewhere, with the reason for each.
- **Consequences:** What the choice buys and what it costs, both kept visible.
- **Choose differently when:** The conditions that would flip the decision for your system.
- **Why it holds up over time:** What keeps the decision right as scale, staff and technology change.
- **Lesson:** The principle that transfers beyond this platform.

## Decision map

**The affinity boundary**: The decision that defines the architecture, and the two that follow from it directly: where the expensive state lives, and what the prompt owes it.

- ADR-01 · The unit of capacity is a warm session; inference is session-affine to the replica holding its prefix
- ADR-02 · The prefix cache is tiered across device, host and local disk, and partitioned by tenant as a hard boundary
- ADR-03 · The prompt's leading segment is contractually stable; volatile context is placed last

**The latency contract**: Who owns the deadline, what is returned when it cannot be met, and where the work is refused rather than slowed.

- ADR-04 · Admission control refuses before generating; a late suggestion is never returned
- ADR-05 · Debounce, single-in-flight and cancellation live in the untrusted client, and enforce nothing
- ADR-06 · Load is shed by refusing in a declared order, never by extending latency

**Measuring a product with no labels**: What counts as better, who is allowed to say so, and how a regression with no error signal is caught at all.

- ADR-07 · Accepted-character retention is the canonical quality metric; acceptance rate is a fast companion
- ADR-08 · The offline evaluation suite is a promotion gate that can only refuse
- ADR-09 · A control arm is held back permanently, because silent quality regression has no error signal

**What may be retained, and what may be suggested**: The two promises an enterprise actually buys: that its code is not kept, and that its own secrets are not handed back to it.

- ADR-10 · Zero retention is implemented as the absence of a path, not as a configuration setting
- ADR-11 · Every filter declares its failure direction; the secret scan is fail-closed inside an 8 ms budget
- ADR-12 · Repository permission is resolved per request against the requesting principal, not cached from index build

**Cost, capacity and the fleet**: Where the money goes in a workload whose majority output is discarded, and how many fleets it takes to hold the latency contract.

- ADR-13 · Cost is measured per accepted suggestion, and wasted generation is a capped, budgeted metric
- ADR-14 · Two models routed by request shape — not one model, and not a deep cascade
- ADR-15 · Each region is a complete working set; prefix caches are never replicated across regions

## Technology by capability

Microsoft Azure was chosen for this exercise for two reasons that point the same way. The first is rotation: across this repository's use cases Amazon Web Services and Google Cloud have dominated the last dozen packages, and Azure has been the least exercised of the three hyperscalers — reaching for the same catalogue repeatedly teaches that catalogue rather than architecture. The second is that this topic genuinely belongs here: the reference implementation of inline code completion at this scale runs on Azure GPU capacity, and the identity, seat and tenancy model that an enterprise rollout depends on is the one most enterprises already operate in Entra ID. The requirement document stays vendor-neutral throughout — "accelerator pool", not a VM SKU — and the table below is the one place where the architecture commits to products. Every row names what would be used instead elsewhere, because the design should survive a change of cloud and this is where that claim is tested.

| Capability | Choice | Origin | Credible alternative | Why this one | Record |
|---|---|---|---|---|---|
| Global edge — anycast termination, WAF, geography pinning, deadline stamping | Azure Front Door Premium with WAF, routing by geography and tenant residency rather than purely by latency | Microsoft Azure | AWS CloudFront with Lambda@Edge; Google Cloud Load Balancing with Cloud Armor; self-hosted Envoy on anycast | The first 60 ms of the budget is spent here, so termination has to be close to the developer and the residency decision has to be made before the request is forwarded anywhere | ADR-04 |
| Inference fleet — continuous batching, session-affine placement | AKS GPU node pools, one pool per model family, with a drain protocol that completes in-flight sequences and refuses new sessions | Microsoft Azure | Amazon EKS with GPU node groups; GKE with GPU node pools; self-operated bare metal with a Kubernetes overlay | Session affinity needs placement control over individual replicas and a drain that is graceful rather than a termination grace period, which rules out an abstraction that hides the node | ADR-01 |
| Prefix cache tiers — device, host, node-local disk | In-process management in the model server, spilling to host DRAM and then to the node's local NVMe temp disk, sized for the working set | open source on Azure | Any serving runtime with paged attention and host offload; a remote cache tier over RDMA where the interconnect justifies it | Eviction has to degrade to a local disk read rather than to a full recompute, which is what holds the cold-prefix penalty at 180 ms | ADR-02 |
| Stateless request plane — context assembly, routing, post-processing, filters | Azure Container Apps, scaled on ordinary request signals, with filters in-process rather than as a network hop | Microsoft Azure | AWS Fargate or App Runner; Google Cloud Run; plain Kubernetes deployments alongside the GPU pools | These tiers hold no state and must scale independently of the accelerator fleet; an in-process filter is the only way to fit an 8 ms p99 budget | ADR-11 |
| Affinity map and policy cache | Azure Cache for Redis, regional, holding (tenant, session) → replica with a session-length TTL and the read-through policy cache | Microsoft Azure | Amazon ElastiCache or MemoryDB; Google Memorystore; self-hosted Redis or Valkey in-cluster | Placement lookup is on the critical path and must answer in single-digit milliseconds; the same store carries the 30-second policy cache so both share one failure mode | ADR-01 |
| Policy, tenancy, seats, consent — the system of record | Azure SQL Database, one write region with regional read replicas, plus an append-only audit store retained five years | Microsoft Azure | Amazon Aurora PostgreSQL; Google Cloud SQL or Spanner where multi-region writes are genuinely needed | Policy must be strongly consistent and is read through a cache, so write latency matters far less than correctness and auditability | ADR-10 |
| Model and decoding-config registry | Azure Container Registry, geo-replicated, content-addressed by digest, with artifacts signed through Key Vault | Microsoft Azure | Amazon ECR with geo-replication; Google Artifact Registry; an OCI registry over object storage | Weights are large, immutable and read in every region; addressing by digest is what makes a 10-minute rollback an alias move rather than a redeploy | ADR-08 |
| Interaction telemetry transport | Azure Event Hubs, two separate streams — code-bearing and non-code — with the code-bearing sink absent for zero-retention tenants | Microsoft Azure | Amazon Kinesis or MSK; Google Pub/Sub; self-hosted Kafka | The split has to be in the transport rather than in a consumer, so that a zero-retention tenant has no configured sink rather than a filtered write | ADR-10 |
| Quality aggregates and experiment analysis | Azure Data Explorer, minute-grain aggregates by model digest, language, recipe version and experiment arm, 400-day retention | Microsoft Azure | Amazon Redshift or Athena over S3; Google BigQuery; ClickHouse self-hosted | Retention and acceptance are computed over high-cardinality interaction events repeatedly and interactively; a columnar engine with fast time-series aggregation is the shape of the query | ADR-07 |
| Identity, seats and short-lived credentials | Microsoft Entra ID, OIDC device-code flow in the editor, 60-minute tokens refreshed by the plugin | Microsoft Azure | Okta or Auth0; AWS IAM Identity Center; Google Cloud Identity — in practice whatever the enterprise already runs | The enterprises adopting this already operate their developer identity here, and seat revocation has to take effect in 60 seconds against the same directory | ADR-05 |
| Retrieval indexes — per workspace, permission-free | Self-operated lexical and symbol indexes on AKS, shards content-addressed in Blob Storage, rebuilt from the repository | open source on Azure | Azure AI Search or Amazon OpenSearch where index count is low; a managed vector store where semantic retrieval dominates | Tens of thousands of per-workspace indexes exceed what a managed search service is priced or shaped for, and the indexes carry no permissions so they can be rebuilt freely | ADR-12 |
| Observability and the operational signals that drive scaling | Azure Monitor with Managed Prometheus and Managed Grafana; admission refusals and queue-wait at p99 as the GPU autoscaling signal | Microsoft Azure | Amazon CloudWatch with Managed Prometheus; Google Cloud Operations; self-hosted Prometheus with Grafana and Mimir | The autoscaler must act on refusals and queue-wait rather than utilisation, which requires custom metrics to be first-class rather than an add-on | ADR-04 |
| Keys, signing and tenant-scoped encryption | Azure Key Vault, with Managed HSM for tenants requiring customer-managed keys and for model-artifact signing | Microsoft Azure | AWS KMS and CloudHSM; Google Cloud KMS; HashiCorp Vault with an HSM backend | Signing the model artifacts is what makes the registry's immutability claim checkable, and customer-managed keys are a standing enterprise requirement | ADR-10 |

## The decisions, and the alternatives that lost

### The affinity boundary

*The decision that defines the architecture, and the two that follow from it directly: where the expensive state lives, and what the prompt owes it.*

#### ADR-01 · The unit of capacity is a warm session; inference is session-affine to the replica holding its prefix

**Status:** Accepted  ·  **Shown on views:** 02, 08, 15, 17

*Between two keystrokes the prompt barely changes. Does the fleet exploit that, and if it does, what is given up to make the exploitation possible?*

**Context.** A completion request's prompt is thousands of tokens of surrounding code. The next request in the same session, sixty milliseconds later, is the same thousands of tokens plus one character. Recomputing attention over the whole prefix on every keystroke is affordable at a demo's traffic and ruinous at 2.4 billion requests a day: it is simultaneously the largest latency cost and the largest money cost in the system. Reusing the cached prefix state is therefore not an optimisation but the economic basis of the product. The difficulty is that the cached state lives in a particular replica's memory, which means the request must go where the state is. That one requirement breaks the assumption every other tier in a modern platform is built on — that any replica can serve any request — and it does so in the tier that is hardest to operate: the one holding four thousand accelerators.

**Decision.** Inference is session-affine. A session identifier, stable for the editing session, is routed to the replica that holds that session's key-value prefix; capacity is planned, scaled and alerted on in units of concurrent warm sessions rather than requests per second. Every other tier — edge, context assembly, post-processing, telemetry ingest — is stateless and disposable. When no replica holds the session, any replica may take it and pay the cold-prefix cost; affinity is a preference with a measured penalty, never a queueing constraint.

**How it is realised on Azure.** AKS GPU node pools run the model servers, with the affinity map held in Azure Cache for Redis as (tenant, session) → replica with a session-length TTL. The affinity router runs in the stateless request plane on Container Apps and resolves placement before admission. Horizontal pod autoscaling is driven by custom metrics — admission rejection rate and queue-wait at p99 — published to Azure Monitor, not by GPU utilisation. Node pools use a drain protocol that stops accepting new sessions while completing in-flight sequences, with the eviction surfaced as an affinity-loss counter rather than as a pod restart.

| Option | Verdict | Reasoning |
|---|---|---|
| Session-affine placement with a measured cold-start fallback | Chosen | Buys an 85% prefix hit rate and the latency and cost that depend on it; pays with placement logic, a drain protocol, and scale-in that evicts live sessions |
| Wholly stateless inference, no prefix reuse | Rejected | Trivial to scale and to drain, and multiplies accelerator spend several times over while putting a full prefix recompute inside a 300 ms budget on every keystroke |
| Stateless inference over a shared remote prefix cache | Deferred | Decouples routing from state, and pays a cache transfer on the critical path plus a store every request must touch; revisit if cross-node interconnect makes the transfer cheaper than the recompute |
| Client-side caching of generated continuations | Right elsewhere | Right for a product with a small, repetitive completion space — a SQL console, a form DSL — and useless where the prefix is the developer's own unique file |

**What it buys**

- An 85% prefix hit rate makes p50 first-token latency 120 ms rather than a full recompute, and takes the dominant term out of the cost model at the same time
- Capacity planning becomes honest: the question "how many developers can this fleet hold" has an answer in warm sessions, which is a quantity operators can reason about
- Cancellation becomes cheap. Evicting a sequence while keeping its prefix warm means the 55% of requests nobody reads cost a batch slot rather than a recompute

**What it costs**

- Scale-in is correctness-adjacent: removing a replica evicts live sessions, so autoscaling must be asymmetric and drains must be graceful rather than terminated
- The inference tier needs placement logic, an affinity store, and a failure mode — affinity loss — that no other tier has
- Multi-region stops being a latency decision and becomes a working-set fragmentation decision (ADR-15)

**Choose differently when.** If the prompt were not incremental — a product where each request's context is unrelated to the last, such as a batch code-explanation service or a one-shot refactoring tool — there would be no warm prefix to route to, and a stateless pool would be strictly better. The decision is justified by the keystroke cadence, and it should be revisited the moment the dominant surface stops being per-keystroke. It would also flip if an interconnect made transferring a prefix cheaper than recomputing it: at that point the shared remote cache becomes the better design, keeping the reuse and giving back painless drains.

**Why it holds up over time.** "Place the request where the expensive state already is" does not depend on today's attention implementation. Any generative decoder keeps per-sequence state, and any system serving an incremental high-frequency editing signal will find the previous request's state is almost all of the next one's. The mechanism may become a compiled state, a learned summary or something unnamed; the decision that clients are placed rather than balanced survives the change. What would not survive is making the cached state authoritative — the moment a prefix cannot be recomputed, affinity loss becomes an error instead of a latency penalty, and the whole design stops degrading gracefully.

> **Lesson.** When the state a request needs is expensive to rebuild and cheap to keep, stop balancing and start placing — and keep the state recomputable, so that losing it costs latency rather than correctness.

#### ADR-02 · The prefix cache is tiered across device, host and local disk, and partitioned by tenant as a hard boundary

**Status:** Accepted  ·  **Shown on views:** 07, 11, 15, 20

*Device memory is the scarcest resource on the fleet and it has to hold both the batch and the cached prefixes. How is it divided, and what happens to a prefix that is evicted?*

**Context.** A replica's accelerator memory is contended three ways: model weights, in-flight batch activations, and cached prefixes. Give more to prefixes and the batch narrows, which lowers throughput; give more to the batch and the hit rate falls, which raises latency. Meanwhile the obvious efficiency win is tempting and dangerous: prefixes are mostly code, much of it shared — the same framework imports, the same license header, the same generated client. A cache keyed purely on content would deduplicate across tenants and raise the hit rate materially. It would also mean one tenant's cached state can be reached through another tenant's request, which converts the fleet's chief efficiency mechanism into its chief isolation risk, in a product whose input is the customer's source code.

**Decision.** The prefix cache is three tiers — accelerator device memory, host DRAM, then node-local NVMe — with eviction cascading downward rather than discarding. The cache key includes the tenant identifier as a hard partition: no cache entry is ever reachable from a request belonging to another tenant, regardless of content equality. Cross-tenant deduplication is refused as a design, not configured off. Device-memory allocation between batch width and prefix residency is a tuned, published parameter with the hit rate and batch occupancy both reported.

**How it is realised on Azure.** Model servers on AKS GPU node pools manage the device tier in-process and spill to host DRAM, then to the node's local NVMe temp disk, which is sized for the working set rather than for logs. Tenant partitioning is a prefix on the cache key, enforced in the server rather than by convention. Tier hit rates (HBM / DRAM / NVMe) and batch occupancy are exported to Azure Monitor as first-class capacity signals, and the device-memory split is a per-model-pool configuration value in the registry's decoding config rather than a deployment flag.

| Option | Verdict | Reasoning |
|---|---|---|
| Three tiers, tenant-partitioned, no cross-tenant deduplication | Chosen | Keeps a re-placed or evicted session on a local disk read rather than a full recompute, and makes the isolation claim structural; gives up a real hit-rate gain |
| Content-addressed cache shared across tenants | Rejected | The largest available efficiency win and an unacceptable one: a cross-tenant read path over customer source code, defended only by the correctness of a cache key |
| Device memory only, discard on eviction | Rejected | Simplest, and makes every eviction a full recompute — which turns ordinary batch pressure into a visible latency incident |
| Remote shared cache tier behind NVMe | Deferred | Would let a drained session recover its prefix on a different node; deferred until the transfer cost is reliably below the recompute cost at p95 |

**What it buys**

- Eviction degrades instead of resetting: a prefix pushed out of device memory is a disk read, not a recompute, which is what keeps the cold-prefix penalty at 180 ms rather than several hundred
- The tenant isolation claim is enforced by the key rather than by a policy, which is the form a security reviewer can actually accept
- The batch-versus-residency split becomes a visible, tunable number with two reported metrics, instead of an emergent property nobody owns

**What it costs**

- A measurable hit-rate gain is given up permanently, and with it some accelerator spend
- Node-local NVMe becomes a sizing requirement on the GPU node pool, which constrains instance selection
- Three tiers means three hit rates to reason about, and a regression in the lowest tier is easy to miss because it only shows up at the tail

**Choose differently when.** In a single-tenant deployment — a sovereign or on-premises installation for one organisation, which is phase 3 of the requirement — the partition buys nothing and content-addressed deduplication becomes straightforwardly correct. The decision is a multi-tenancy decision, not a caching one. It would also change if device memory stopped being the binding constraint: with an order of magnitude more of it, the tiering below device memory would matter much less and could be dropped for simplicity.

**Why it holds up over time.** "Efficiency mechanisms that cross a trust boundary are not efficiency mechanisms" is the durable half of this record. The specific tiers will be replaced — memory hierarchies change every accelerator generation, and the names HBM, DRAM and NVMe will not last — but the rule that a cache key carries the tenant, and that sharing is refused rather than configured, will still be the right call when the hierarchy has four levels and different names.

> **Lesson.** Put the trust boundary in the cache key, not in the policy that guards the cache. A deduplication you refuse to build cannot be enabled by a mistake.

#### ADR-03 · The prompt's leading segment is contractually stable; volatile context is placed last

**Status:** Accepted  ·  **Shown on views:** 07, 13, 14

*Prompt composition is usually treated as a quality question. What happens to the cache when it is treated only as a quality question?*

**Context.** Context assembly has freedom in what it includes and in what order. Treated as a relevance problem, the natural instinct is to put the most relevant material closest to the cursor — retrieved snippets, the most recently edited region, whatever the ranker scored highest. All of those change between keystrokes. A prefix cache is valid only up to the first token that differs, so a single volatile item placed early invalidates the entire cached prefix on every keystroke, and the 85% hit rate the whole architecture depends on collapses to near zero. The failure is invisible in testing: quality metrics look fine, latency looks fine at low traffic, and the cost and tail latency only appear at scale. This makes prompt ordering an architectural constraint masquerading as an authoring preference.

**Decision.** The prompt has a declared two-part structure: a stable leading segment — the file's identity, the language preamble, the imported-symbol block, the retrieved repository snippets selected for the session rather than for the keystroke — followed by a volatile tail holding the cursor neighbourhood. Sources are assigned a priority that determines both truncation order and position, and the lowest-priority source is dropped first with the drop recorded. The ordering rule is versioned as a prompt recipe, and the recipe version is logged against every request so a quality change is attributable to composition rather than to the model.

**How it is realised on Azure.** The prompt assembler runs in the stateless request plane on Container Apps; the recipe is versioned configuration in the Azure SQL control plane, read through the same bounded-staleness Redis cache as policy, so a recipe change takes effect within 30 seconds without a deployment. Retrieved snippets are selected once per session and cached with the session rather than re-ranked per keystroke. The recipe version is a column on suggestion_request in the telemetry model; prefix stability is exported as its own metric so that a recipe change that hurts the cache is visible within minutes rather than in next month's bill.

| Option | Verdict | Reasoning |
|---|---|---|
| Declared stable segment plus volatile tail, versioned recipe | Chosen | Protects the hit rate and makes quality attributable; constrains what the ranker may do and when it may do it |
| Relevance-ordered prompt, most relevant nearest the cursor | Rejected | The intuitive design, and it invalidates the cache on every keystroke — a quality decision that silently spends the latency budget it was supposed to help |
| Per-keystroke re-ranking with cache-aware reordering | Deferred | Could get both if the reorderer could prove stability; deferred because it adds a correctness obligation to a component on the critical path |
| Fixed prompt template with no retrieval at all | Right elsewhere | Right for a narrow DSL or a single-file scripting surface, where open tabs and repository context add nothing worth a token |

**What it buys**

- The hit rate the architecture depends on is protected by a rule rather than by hoping, and prefix stability has a metric that fails loudly
- A quality regression can be traced to a recipe version without retaining a single prompt, which keeps attribution compatible with zero retention
- Retrieval becomes a per-session cost rather than a per-keystroke one, which also removes it from the critical path at the frequency that matters

**What it costs**

- The ranker cannot act on the most recent keystroke, so some genuinely better context is excluded by position
- Per-session snippet selection goes stale within a long editing session, and refreshing it is a deliberate cache-invalidating event that has to be scheduled
- Prompt authoring becomes a constrained activity with a review step, which slows down the experiments that quality work depends on

**Choose differently when.** If prefix reuse were abandoned (see the flip on ADR-01), this constraint evaporates and prompt order becomes purely a relevance question. Short of that, it would loosen if models became robust to context position in a way that let the volatile material sit anywhere — but that is a claim about relevance, not about caching, and the cache would still prefer the stable ordering.

**Why it holds up over time.** The transferable property is that a cache's hit rate is a function of the request's construction, not of the cache's implementation — so whoever composes the request owns the hit rate whether they know it or not. That will be true of any prefix-reusing system. Making the composition a versioned, measured artefact rather than code is what keeps it true as the team that writes prompts changes.

> **Lesson.** If a cache keys on a prefix, the order of what you put in the prefix is an architectural decision. Hand it to whoever owns the latency budget, not to whoever owns relevance.

### The latency contract

*Who owns the deadline, what is returned when it cannot be met, and where the work is refused rather than slowed.*

#### ADR-04 · Admission control refuses before generating; a late suggestion is never returned

**Status:** Accepted  ·  **Shown on views:** 02, 13, 15

*When the fleet cannot serve a request inside its deadline, what does the developer get?*

**Context.** The usual answer in a serving system is to queue: accept the work, do it as fast as possible, and let latency degrade under load. That answer is wrong here, because the value of a suggestion falls to zero and then goes negative. A suggestion that arrives after the developer has typed past it is not a slow success — it is grey text appearing over a line they have already written, which costs attention and offers nothing. Worse, under burst the queueing design spends accelerator seconds on suggestions nobody will read, so the fleet is most wasteful exactly when it is most contended. The 55% baseline cancellation rate means this is not an edge case: the system's normal state is that most work in flight is already doomed.

**Decision.** Every request carries the client's remaining deadline. An admission gate compares that deadline against current queue depth and estimated generation time and refuses immediately — before any accelerator work begins — anything it predicts will miss. A refused request returns an explicit empty result, which the client treats as "the model had nothing to say". Nothing is queued without bound, and no result is returned after its deadline has passed, even if it is ready.

**How it is realised on Azure.** The deadline is stamped at the Front Door edge after identity and quota, propagated as a request header through the Container Apps request plane, and evaluated by the admission gate immediately before placement. The gate's inputs are the replica's in-flight sequence count, measured queue-wait at p99 from Azure Monitor, and a per-model generation-rate estimate from the registry's decoding config. Refusals are a first-class metric and the primary autoscaling signal for the GPU node pools (ADR-01); the client translates an empty result into no ghost text and widens its own debounce on repeated refusals.

| Option | Verdict | Reasoning |
|---|---|---|
| Deadline-aware admission, refuse at the door | Chosen | Zero accelerator cost for work that cannot land, and a load signal that is countable; requires predicting generation time before generating |
| Queue and serve as fast as possible | Rejected | The default in most serving stacks, and here it spends the most accelerator seconds on the least valuable work precisely under burst |
| Client-side abandonment only, server finishes the work | Rejected | Simplest contract and strictly worse economics: the server cannot tell a cancelled request from a live one and pays for both |
| Return the best partial result at the deadline | Deferred | Attractive for multi-line suggestions and risky: a truncated multi-line completion can be more misleading than none; revisit once displayability can be judged mid-stream |

**What it buys**

- A 4× burst becomes a capacity question with a number attached — the refusal rate — rather than a latency incident with a shrug attached
- Accelerator seconds are not spent on suggestions that cannot be displayed, which is what makes the cost-per-accepted target reachable
- The refusal rate is a leading autoscaling and alerting signal, which average GPU utilisation can never be

**What it costs**

- Generation time must be estimated before generation, from features available at admission; a bad estimator either refuses work it could have served or admits work it cannot
- Developers in a contended region experience the product as intermittently absent, which is a silent degradation they cannot report usefully
- The empty result is indistinguishable from "the model genuinely had nothing", so the refusal rate has to be instrumented server-side or the degradation is invisible

**Choose differently when.** If the product's value did not decay inside a second — a batch completion pass over a file, a pull-request suggestion run, an agentic edit the developer waits for — queueing would be correct and refusing would be perverse. The decision belongs to the keystroke surface specifically. It would also soften if accelerator capacity became cheap enough that peak could simply be provisioned for, at which point the gate still exists but almost never fires.

**Why it holds up over time.** Deadline propagation plus refuse-rather-than-queue is a protocol property, not a framework feature, and it is right wherever the value of a response decays faster than the queue drains. It has outlasted several generations of serving infrastructure already. The part that would date is the estimator, which depends on the decoder; the part that will not is making the refusal countable, because that is what lets capacity be planned honestly.

> **Lesson.** If a response loses its value with age, admission control is not back-pressure — it is the feature. Refuse early, count the refusals, and never deliver something that has already expired.

#### ADR-05 · Debounce, single-in-flight and cancellation live in the untrusted client, and enforce nothing

**Status:** Accepted  ·  **Shown on views:** 01, 09, 13, 20

*The editor is the one component the platform cannot control and the only one that knows the keystroke. How much of the latency contract is allowed to depend on it?*

**Context.** Three of the mechanisms that make this system affordable can only exist in the editor. Debounce requires knowing that a keystroke happened and that no further keystroke followed for sixty milliseconds. Cancellation requires knowing that the developer kept typing. Suppression inside string literals, comments and already-complete lines requires the editor's own parse of the buffer. None of these can be done server-side at this budget, because the information arrives with the keystroke and the round trip is the thing being avoided. But the editor plugin is also the least trustworthy component in the system: it runs on the developer's machine, it ships on its own release cadence, and a single bad version can change fleet economics by tens of percent with nothing server-side having changed.

**Decision.** The client owns debounce, single-in-flight-per-session, cancel-on-keystroke and local suppression, and these are published as a versioned client contract with a supported version floor. No policy decision, filter, quota or authorisation is enforced only on the client: every one of them is independently enforced server-side, so a modified or defective client can waste capacity and cannot gain access or bypass a filter. Clients identifying a version below the floor are refused rather than served degraded, and a server-side kill switch can force any client version into a reduced mode without a client release.

**How it is realised on Azure.** The contract is versioned alongside the Suggestion API and enforced at the Front Door edge, which reads the client version header and consults the flag service before anything else. The version floor, the reduced-mode flag and the per-tenant and per-developer kill switches live in the Azure SQL control plane and reach the edge through the policy cache, taking effect for new requests within 30 seconds. Per-seat rate limiting at the edge bounds what a misbehaving client can cost; cancellation-rate distribution is reported per client version so a bad release is visible as a distribution shift rather than as an unexplained spend increase.

| Option | Verdict | Reasoning |
|---|---|---|
| Client owns the timing mechanisms; server enforces everything that matters | Chosen | The only arrangement that fits the budget and survives a hostile client; accepts that capacity cost is partly outside the platform's control |
| Server-side debounce and suppression | Rejected | Requires the round trip that debounce exists to avoid, and still cannot see the keystroke that should have cancelled the request |
| Trust the client for quota and suppression as well | Rejected | Cheaper and indefensible: a plugin is a program on someone else's machine, and a filter it can skip is not a filter |
| A thin local daemon owned by the platform, with the editor as a shell | Deferred | Gives the platform a controlled client without the round trip; deferred because it multiplies the install and update surface across four editors and three operating systems |

**What it buys**

- The timing mechanisms sit where the information is, which is what makes the 300 ms budget attainable at all
- Authorisation, policy and filtering remain enforceable against a client the platform does not control, so adoption does not rest on client integrity
- The version floor and reduced mode give an operator two blunt but immediate levers when a release goes wrong

**What it costs**

- Accelerator spend is partly determined by client behaviour the platform can only observe after the fact
- Version skew is permanent across four editors, so the contract must stay backward-compatible for longer than is comfortable
- Every policy must be implemented twice — once in the client for latency, once on the server for truth — and the two can drift

**Choose differently when.** If the platform shipped the editor itself, the client would stop being untrusted and much of the duplication could go. That is the situation for a vendor whose own IDE is the only surface, and their design would legitimately differ. It would also flip if a server round trip got cheap enough — on a local network, or for an in-browser IDE served from the same region — that debounce could move server-side.

**Why it holds up over time.** "Put the mechanism where the information is; put the enforcement where the trust is" is the durable statement, and it will survive every editor, protocol and plugin API in this space. What dates is the specific list of mechanisms: a future client may own more of the pipeline, or less, but the split between latency-driven placement and trust-driven enforcement stays.

> **Lesson.** A component can be load-bearing and untrusted at the same time. Let it carry the timing it alone can see, and never let it carry a decision you would have to defend.

#### ADR-06 · Load is shed by refusing in a declared order, never by extending latency

**Status:** Accepted  ·  **Shown on views:** 02, 15, 18

*When there is not enough capacity for everything, who loses their suggestion first — and is that a decision or an accident?*

**Context.** Admission control (ADR-04) decides whether a single request can make its deadline. It does not decide which requests should be sacrificed when the fleet as a whole is short. Without an explicit answer, the answer emerges from queue mechanics: whoever happens to arrive during the spike loses, which means the loss is distributed by luck and is impossible to explain to a tenant afterwards. Worse, the most expensive work — multi-line suggestions, speculative prefetch — survives at the same rate as the cheapest, so shedding recovers far less capacity than it costs in goodwill. A fleet serving 1.2 million developers with a 5× trough-to-peak daily swing will be short regularly enough that this needs to be a design rather than an emergent property.

**Decision.** Shedding is ordered and published: speculative prefetch is shed first, then multi-line suggestions, then lower service tiers, and single-line user-initiated requests on the highest tier are shed last. Shedding is always a refusal — an explicit empty result — and never an extension of latency. A tenant that has reached a spend ceiling degrades along the same declared ladder (smaller model, single-line only, then suspension) rather than being cut off without warning.

**How it is realised on Azure.** The shedding order is configuration in the Azure SQL control plane, read through the policy cache and evaluated by the admission gate alongside the deadline check. Request shape (single-line versus multi-line) is already known at admission because it determines model routing (ADR-14), so no extra classification step is needed. Shed counts are exported per class and per tenant to Azure Monitor, and the per-tenant degradation ladder is surfaced in the admin API so an administrator sees which rung they are on before they see the bill.

| Option | Verdict | Reasoning |
|---|---|---|
| Published shedding order, refusal only | Chosen | Recovers the most capacity per unit of lost value and makes degradation explainable; requires classifying request value at admission |
| Fair-share shedding across tenants | Rejected | Equitable and inefficient: it protects cheap and expensive requests equally, so it sheds more requests to recover the same capacity |
| Latency degradation under load | Rejected | The implicit default, and the one outcome this product cannot absorb — a late suggestion is worse than no suggestion |
| Priority bidding per tenant | Right elsewhere | Right where customers genuinely value latency differently and will pay for it; here it would make the product's core experience a purchasable variable, which is a commercial decision rather than an architectural one |

**What it buys**

- Degradation becomes a decision with an owner and a published order, which is the difference between a managed shortage and an incident
- Shedding the expensive classes first recovers more accelerator capacity per refused request, so the shortage ends sooner
- Spend-ceiling behaviour and capacity-shortage behaviour share one ladder, so there is one degradation story rather than two

**What it costs**

- Multi-line suggestions are structurally the least reliable feature of the product, which is a permanent asymmetry users will notice
- Request-class value is a judgement encoded in configuration, and a wrong ordering is only discoverable from acceptance data afterwards
- Tier-based shedding couples the architecture to the commercial packaging, which then becomes hard to change

**Choose differently when.** If capacity were elastic on the timescale of a burst — accelerators available in seconds rather than minutes — shedding would matter far less and provisioning would do the work instead. It would also change if multi-line suggestions turned out to carry most of the measured retention, in which case the ordering is simply wrong and the data should overturn it.

**Why it holds up over time.** "Shed by value, in an order you published before the incident" outlives every scheduler. The specific classes here are tied to this product's shapes, but the principle — that a shortage should be resolved by a decision made in advance rather than by queue mechanics in the moment — is one of the few operational rules that has never stopped being true.

> **Lesson.** Decide who loses before you are short. An unplanned shortage is distributed by luck, and luck is the one allocation policy nobody can defend afterwards.

### Measuring a product with no labels

*What counts as better, who is allowed to say so, and how a regression with no error signal is caught at all.*

#### ADR-07 · Accepted-character retention is the canonical quality metric; acceptance rate is a fast companion

**Status:** Accepted  ·  **Shown on views:** 04, 16, 18, 19

*Nobody tells this system whether a suggestion was good. What stands in for that, and what does the substitute get wrong?*

**Context.** The available signals are all behavioural. A suggestion is displayed, or cancelled before display; if displayed it is accepted or ignored; if accepted it survives or is deleted. Acceptance rate is the obvious metric: it is immediate, abundant and easy to compute. It is also systematically misleading in the one direction that matters. A model that is more confident and more fluent produces suggestions that look right and get taken, and some fraction of those are deleted twenty minutes later when the developer discovers the API does not exist. Optimising acceptance therefore optimises plausibility, which is precisely the failure mode of the underlying technology. Retention — how much of an accepted suggestion still exists in the file after a delay — sees the difference, and arrives a day late.

**Decision.** Accepted-character retention is the canonical quality metric, measured at 120 seconds and again at 24 hours after acceptance. Acceptance rate and time-to-first-token are fast guard rails that may halt a rollout on their own but may never promote one. A model that improves acceptance and regresses retention is refused. Both metrics are always reported against a control arm (ADR-09) rather than against the model's own history.

**How it is realised on Azure.** The editor plugin probes the accepted span at 120 seconds and reports surviving characters as an asynchronous telemetry event to Event Hubs; the 24-hour measurement is derived from a later session's buffer state for the same span. Events land in Azure Data Explorer, where retention and acceptance are materialised per model digest, language, prompt recipe version and experiment arm. The promotion gate reads those aggregates; thresholds live in the control plane so a gate can be tightened without a deployment. Only character counts and span offsets are transmitted — never the characters themselves — which is what keeps the metric compatible with zero retention.

| Option | Verdict | Reasoning |
|---|---|---|
| Retention canonical, acceptance as a guard rail | Chosen | Measures usefulness rather than plausibility; costs a day of latency in the decision loop and a client-side probe |
| Acceptance rate as the primary metric | Rejected | Immediate, abundant, and rewards exactly the failure mode of a confident model; this is the industry default and it is the wrong default |
| Explicit developer feedback (thumbs up/down) | Rejected | Sparse, biased toward the annoyed, and an interruption in a surface whose entire value proposition is not interrupting |
| Downstream signals — test pass rate, review comments, revert rate | Deferred | The truest measure of usefulness and far too slow and confounded to gate a weekly model cadence; worth building as a quarterly check |

**What it buys**

- The metric that gates promotion measures whether the suggestion was useful rather than whether it was convincing, which is the only defence against optimising for plausibility
- Retention is computable from character counts and offsets alone, so quality measurement survives a zero-retention tenant
- Per-language and per-recipe retention makes it possible to see a model that improved the mean and ruined one language

**What it costs**

- A one-day decision latency, which bounds how fast models can be iterated and forces a capped canary in the meantime
- The probe depends on the client, so a plugin release can move the headline quality metric without the model changing
- A span that was accepted and then legitimately refactored counts as lost retention, which puts a noise floor under the metric

**Choose differently when.** If the product moved from inline completion to whole-function or agentic edits, the unit of measurement changes — a diff accepted and then reworked is normal rather than a failure — and retention at the character level stops being meaningful. At that point test outcomes and review signals become the right instrument despite their latency.

**Why it holds up over time.** Every offline benchmark eventually saturates or gets trained against. What does not saturate is whether a human kept the output. The question retention asks — did this survive contact with the work — is independent of what the model can do, so it will still be the right instrument when the model writes whole modules. The 120-second probe is an implementation detail; the choice of a survival metric over an uptake metric is not.

> **Lesson.** When you cannot measure correctness, measure survival. Uptake tells you the output was persuasive; survival tells you it was right, and only one of those is worth optimising.

#### ADR-08 · The offline evaluation suite is a promotion gate that can only refuse

**Status:** Accepted  ·  **Shown on views:** 05, 16, 19

*An offline suite over held-out completion contexts is cheap and fast. Why is it not allowed to decide anything?*

**Context.** Offline evaluation is attractive because it is fast, repeatable and available before any developer is exposed. The problem is that its correlation with retention is weak and unstable: exact-match and edit-distance scores reward a model that reproduces the held-out continuation, which is not the same as producing something the developer keeps, and the correlation shifts whenever the corpus, the languages or the model family changes. Treating such a score as a verdict means putting a model in front of 1.2 million developers on the strength of a proxy whose relationship to the real metric nobody can state. Treating it as nothing, however, wastes the one check that can catch an obviously broken candidate for a few dollars.

**Decision.** The offline suite is a gate with one direction of authority: it can refuse a candidate before it reaches a canary, and it can never promote one. A model that has not been evaluated cannot be promoted at all. Promotion requires an online retention verdict (ADR-07) measured against the control arm. Offline thresholds are set to catch gross failures — wrong language, degenerate output, collapse on a common construct — rather than to rank near-equivalent candidates.

**How it is realised on Azure.** The harness runs as a job against a held-out corpus built only from licensed material, stored in Blob Storage and versioned alongside the registry. A model version in Azure Container Registry carries an eval_run_id, and the Model API refuses a traffic split for any digest without one. Thresholds are control-plane configuration, so tightening the gate is a configuration change and not a pipeline release. Results are retained for two years so that a later retention surprise can be checked against what the offline suite said at the time — which is how the correlation gets re-estimated rather than assumed.

| Option | Verdict | Reasoning |
|---|---|---|
| Offline as a refusing-only gate, retention as the verdict | Chosen | Catches broken candidates cheaply without granting authority to a weak proxy; costs a day of canary before anything can ship |
| Offline score as the promotion criterion | Rejected | Fast, cheap, and the most common way a completion product regresses — it promotes the model that best reproduces the corpus, not the one developers keep |
| No offline evaluation; canary everything | Rejected | Exposes developers to candidates a mechanical check would have caught, and burns canary capacity that is the scarce resource in the loop |
| Human evaluation panels on sampled completions | Right elsewhere | Right for a lower-volume, higher-stakes surface; at a weekly cadence across dozens of languages the panel becomes the bottleneck and its judgement does not generalise |

**What it buys**

- An obviously broken candidate is stopped for the price of a batch job, before any developer is exposed
- Nothing is promoted on a metric whose relationship to usefulness cannot be stated, which is the whole point of the asymmetry
- Keeping offline results alongside later retention outcomes turns the correlation into something measured rather than assumed

**What it costs**

- Every candidate pays a canary day, even one that is obviously better offline
- Near-equivalent candidates cannot be ranked before exposure, so canary capacity rather than compute becomes the limit on iteration speed
- A licensed-only corpus is narrower than production traffic, so the gate is blind to constructs that only appear in customers' code

**Choose differently when.** If an offline metric were ever shown to predict retention reliably across languages and model families, the asymmetry could be relaxed and promotion could move offline — which would transform iteration speed. That is a measurement result, and the retained eval history exists precisely so it could one day be demonstrated rather than claimed.

**Why it holds up over time.** The durable statement is about the direction of authority, not about the metric: a cheap proxy may veto and may not decide. That rule holds for every proxy in every system, and it is what prevents proxy metrics from quietly becoming objectives. It will survive the replacement of exact-match scoring by whatever succeeds it.

> **Lesson.** Give a cheap proxy the power to refuse and never the power to approve. The asymmetry is what stops a proxy becoming the objective.

#### ADR-09 · A control arm is held back permanently, because silent quality regression has no error signal

**Status:** Accepted  ·  **Shown on views:** 16, 18, 19

*Which failure in this system produces no error, no alert and no page — and what instrument detects it?*

**Context.** Every other failure class here announces itself. Latency misses show up in a percentile, saturation in admission refusals, dependency failure in a degradation counter, filter failure in a suppression count. Quality is different: a model that has become less useful looks exactly like a healthy fleet. Latency is fine, errors are zero, acceptance drifts by a percent and nobody notices for a month, by which time it is attributed to a tooling change or a seasonal shift. Comparing against the model's own history does not help, because the population drifts too — new tenants, new languages, a change in the mix of work. The only reliable reference is a concurrent population served by a known-good configuration.

**Decision.** A control arm is held back permanently, not per experiment: a stable, randomly assigned fraction of developers continue on the previous known-good model and prompt recipe. All quality figures are reported as differences against that arm. The gate that halts a rollout reads the difference, not the absolute. Measurement failures — a telemetry gap, a broken probe, an arm whose assignment has drifted — are alerted separately from quality regressions, so an on-call engineer can tell a broken instrument from a worse model.

**How it is realised on Azure.** Arm assignment is stable per developer and held in the Azure SQL control plane as experiment_arm rows with an is_control flag, so an assignment survives restarts, plugin upgrades and cache flushes. Aggregates in Azure Data Explorer are computed per arm and per language. The control arm runs on its own share of accelerator capacity with the previous model mounted, which is capacity permanently unavailable to the current one. Arm-balance and telemetry-coverage checks run continuously and page a different rotation from quality alerts.

| Option | Verdict | Reasoning |
|---|---|---|
| Permanent held-back control arm, all figures as differences | Chosen | The only instrument that distinguishes a worse model from a changed population; costs permanent capacity and leaves some developers on an older model |
| Per-experiment control, no standing arm | Rejected | Cheaper and blind between experiments, which is exactly when a slow regression accumulates unnoticed |
| Compare against the model's own history | Rejected | Confounded by population drift, seasonality and plugin releases — it answers a different question from the one being asked |
| Synthetic canary traffic against a fixed probe set | Deferred | Useful for serving health and useless for quality, because a synthetic developer never accepts or deletes anything; worth keeping as an availability probe only |

**What it buys**

- The one failure class with no error signal acquires an instrument, and an automated rollback can act on it without a human noticing first
- Population drift stops being confounded with model quality, which makes a 1.5% effect claim defensible
- Separating measurement-failure alerts from quality alerts removes the worst minute in the system: not knowing whether the fleet or the thermometer is broken

**What it costs**

- A permanent slice of developers gets a deliberately older model, which is a real and ongoing product cost
- Control-arm capacity holds the previous model warm forever, taking working set from the current one (ADR-15)
- Permanent assignment creates a long-lived population that must be rebalanced carefully, because a drifted arm silently invalidates every figure computed against it

**Choose differently when.** If quality could be measured absolutely — a reliable automated correctness check on each suggestion — the control arm's main job would disappear and it could shrink to an availability probe. Short of that, the arm shrinks only if the developer population stops drifting, which for a growing product it will not.

**Why it holds up over time.** "A metric with no error signal needs a concurrent reference, not a historical one" is the transferable half, and it applies far beyond this product — to recommendation quality, ranking relevance and anything else judged by behaviour. The cost of the arm will fall as capacity gets cheaper; the need for it will not change at all.

> **Lesson.** For any failure that produces no error, buy a concurrent control rather than a better alert. You cannot alert on a number you have nothing to compare against.

### What may be retained, and what may be suggested

*The two promises an enterprise actually buys: that its code is not kept, and that its own secrets are not handed back to it.*

#### ADR-10 · Zero retention is implemented as the absence of a path, not as a configuration setting

**Status:** Accepted  ·  **Shown on views:** 01, 10, 11, 20

*An enterprise security reviewer is told retention is disabled. Why should they believe it, and what would make the claim checkable?*

**Context.** The input to this system is the customer's source code, which is usually their most valuable asset and often the one with the strictest contractual controls. A reviewer evaluating adoption knows that a setting can be misconfigured, re-enabled by a deployment, bypassed by a new code path, or quietly contradicted by infrastructure nobody considered — crash dumps, request logs, distributed traces, a debugging flag left on in one region. They are right to discount a configuration claim, because the history of such claims is bad. Meanwhile the platform genuinely needs telemetry: without interaction data there is no quality measurement at all (ADR-07, ADR-09), and without quality measurement the product cannot improve.

**Decision.** Telemetry is split at the boundary into a code-bearing stream and a non-code stream, and for a zero-retention tenant the code-bearing path does not exist: no sink is configured, no buffer is allocated, and the scrub is enforced at the sink as well as at the source so that infrastructure which does not know about retention mode cannot create one. Zero retention is the default for enterprise tenants. Diagnostic retention is opt-in, bounded at 24 hours, and drawn in red on the data-flow view because it is the only route by which tenant code reaches durable storage. There is no data path at all from tenant traffic to model training, and evaluation corpora are built from licensed material only.

**How it is realised on Azure.** The post-processing tier emits two Event Hubs streams; the code-bearing stream has no configured sink for a zero-retention tenant, so the emit is a no-op rather than a filtered write. A scrubber enforces the same rule at the Data Explorer and Blob sinks, which is where crash dumps and trace exporters would otherwise land. Retention mode is a column on the tenant row in Azure SQL, read through the policy cache, and failing closed to zero retention when the cache is cold. Consent and configuration changes are written to an append-only audit store kept five years, so a reviewer can see when the mode changed and who changed it.

| Option | Verdict | Reasoning |
|---|---|---|
| Absent path for code-bearing telemetry, enforced at source and sink | Chosen | A control that cannot be misconfigured and does not drift; costs the ability to diagnose a tenant-specific quality problem from data |
| Retention flag checked before writing | Rejected | The common design, and the one a reviewer is right to discount: one new code path, one debug flag or one unaware exporter defeats it |
| Retain with encryption and customer-held keys | Rejected | Strong on paper and still retention: the customer's code is in the platform's storage, which is the fact the review is actually about |
| Retain anonymised or tokenised code | Rejected | Source code is not anonymisable in any meaningful sense — identifiers, structure and comments carry the identity, and a fragment is often enough to identify the repository |

**What it buys**

- The privacy claim becomes structural rather than procedural, which is the form that survives an audit and a refactor
- Quality measurement still works, because retention and acceptance need character counts and offsets rather than characters
- A cold policy cache fails to zero retention, so an availability problem can never become a disclosure

**What it costs**

- A tenant-specific quality complaint cannot be investigated from retained examples, only from aggregates and a reproduction
- Evaluation corpora stay narrower than production traffic, so the offline gate is blind to constructs that only appear in customer code (ADR-08)
- Two telemetry streams and two enforcement points is more machinery than one flag, and the sink-side scrub must be re-verified whenever infrastructure changes

**Choose differently when.** A tenant that actively wants retention — to debug its own usage, or to build an internal corpus under its own legal basis — can opt in, and that is the diagnostic path. For a single-tenant sovereign deployment the whole question changes shape, because the retaining storage is the customer's own. The decision is about a shared multi-tenant platform holding other people's source code.

**Why it holds up over time.** A control implemented as a missing code path cannot be misconfigured, cannot drift, and does not need re-attesting after every refactor. As regulation around training data and code provenance tightens, the designs still adoptable will be the ones where the sensitive path was never built. That argument gets stronger with time, not weaker.

> **Lesson.** The strongest version of a privacy guarantee is an absent capability. If a path exists, you are promising about a configuration; if it does not, you are describing the system.

#### ADR-11 · Every filter declares its failure direction; the secret scan is fail-closed inside an 8 ms budget

**Status:** Accepted  ·  **Shown on views:** 02, 08, 18

*A filter on a 300 ms critical path will eventually be slow or unavailable. What happens to the suggestion then, and who decided?*

**Context.** Two filters sit between generation and display. A secret scan catches keys, tokens and private-key blocks that the model may reproduce from its context or its training — the failure where a developer is handed back a credential, sometimes one of their own. A public-code match filter suppresses suggestions that reproduce a substantial verbatim span of known public source, which some enterprises treat as a licence exposure and others ignore. Both run inside a budget measured in single-digit milliseconds. In most systems what happens when such a filter is slow or down is never decided: it times out, and whichever way the timeout happens to be coded becomes the policy. For one of these filters failing open is a security incident and for the other it is a commercial preference, so a single timeout behaviour cannot be right for both.

**Decision.** Each filter declares its failure direction explicitly as part of its configuration, and the choice is a product decision rather than an implementation detail. The secret scan is fail-closed: if it cannot run, no suggestion is displayed. The public-code match filter is fail-closed for tenants that selected block mode and fail-open for tenants in warn or off mode. Every suppression is counted and attributed, so suppression is visible rather than silent, and a filter that starts suppressing a large fraction of candidates is an alert rather than a mystery.

**How it is realised on Azure.** Filters run in the stateless post-processing tier on Container Apps, in-process rather than as a network hop, so the budget is reachable. Failure direction, mode and thresholds are per-tenant configuration in Azure SQL read through the policy cache, defaulting to the stricter setting when the cache is cold. Filter time is reported separately from inference time in Azure Monitor. The public-code match index is refreshed in batch from a licensed corpus in Blob Storage and is never on the critical path for a tenant in off mode. Suppression counts by filter, tenant and reason go to Data Explorer alongside the interaction telemetry.

| Option | Verdict | Reasoning |
|---|---|---|
| Per-filter declared failure direction, counted suppressions | Chosen | Makes the safety-versus-availability trade explicit per filter and per tenant; costs configuration surface and two code paths per filter |
| One uniform timeout policy for all filters | Rejected | Simple and wrong for at least one filter by construction: the right answer differs between a credential and a licence preference |
| Filter asynchronously, retract after display | Rejected | A suggestion already displayed is a suggestion already read; retraction after the fact does not undo a disclosed secret |
| Prevent at decoding time with constrained generation | Deferred | Removes the post-hoc filter and its latency, and costs tokens-per-second on the critical path and a much harder correctness argument; worth revisiting as constrained decoding matures |

**What it buys**

- The one failure that cannot be allowed — handing back a credential — is structurally prevented rather than left to a timeout's default
- Tenants with different licence postures get different behaviour without different code paths in the filter itself
- Counted suppressions turn a silent quality cost into a measurable one, which is what makes the false-positive budget enforceable

**What it costs**

- A secret-scan outage is a full product outage for every tenant, which makes that component as availability-critical as inference
- Filtering spends part of a very small budget, and the spend is on the path of every displayed suggestion
- Each filter needs two tested code paths and a per-tenant configuration, which is a permanent increase in surface

**Choose differently when.** If generation-time prevention matured enough to make verbatim reproduction of secrets and licensed spans structurally impossible, the post-hoc filters could shrink to an audit sample and the fail-closed coupling would disappear. For a tenant with no licence concern and no secrets in its repositories — rare, and usually wrong about both — the filters could run in sample mode.

**Why it holds up over time.** "Every control on a critical path declares which way it fails, and the declaration is a product decision" is a rule that outlives any particular filter. It is also one of the few security properties that gets cheaper to keep as a system grows, because it is enforced by configuration shape rather than by vigilance.

> **Lesson.** Decide a control's failure direction when you build it, not when it fails. An undeclared timeout is a policy chosen by whoever wrote the error handler.

#### ADR-12 · Repository permission is resolved per request against the requesting principal, not cached from index build

**Status:** Accepted  ·  **Shown on views:** 14, 20, 21

*A retrieval index is built once and read many times. Whose permissions does it serve — the builder's, or the reader's at the moment they read?*

**Context.** Retrieval over the wider repository is what lifts a completion from the current file to the codebase, and it is the feature most likely to cause a disclosure. An index built under one set of permissions outlives those permissions: a contractor leaves a project, a repository is made private, a monorepo path moves behind an access list. If permission is captured at build time, the index keeps serving the old answer until it is rebuilt, and rebuilds take up to an hour for the largest workspace. The alternative is to resolve permission per request against the system that actually owns it, which puts a synchronous call to a third party inside a 300 ms budget — on a path where the obvious optimisation, caching the answer, is also the exact mistake being avoided.

**Decision.** Repository permission is evaluated at request time against the requesting principal, using the source hosting system as the authority. The default retrieval scope is the workspace, not the tenant. When the permission authority cannot answer inside the assembly budget, retrieval degrades to open-tabs-only context and the degradation is counted as a quality signal rather than treated as a silent fallback. Index contents are never served on the basis of permissions recorded at build time.

**How it is realised on Azure.** The retrieval service in the request plane calls source hosting with the principal from the validated Entra token and receives a scoped repository list, then retrieves only within that scope from the per-workspace indexes. Short-lived negative results may be cached; positive grants are not cached beyond the request. Private endpoints carry the call, and its latency is budgeted inside the server-side 15 ms assembly allowance with the degradation path taking over beyond it. Degradation-to-open-tabs is exported per tenant to Azure Monitor and correlated with acceptance in Data Explorer, so the cost of the degradation is visible in quality rather than only in latency.

| Option | Verdict | Reasoning |
|---|---|---|
| Per-request resolution against the owning authority, degrade on timeout | Chosen | Revocation takes effect immediately and the index cannot outlive a permission; costs a third-party dependency inside the critical path |
| Permissions captured at index build | Rejected | Fast, cheap and wrong by up to a rebuild cycle — which is exactly the window in which a revocation matters most |
| Per-principal index partitioning | Rejected | Removes the request-time call and multiplies index count by the number of developers, which at 1.2 million makes the storage and rebuild cost absurd |
| Cached grants with a short TTL and push invalidation | Deferred | The pragmatic middle, and it depends on an invalidation signal from a system the platform does not own; revisit if source hosting offers a reliable one |

**What it buys**

- A revocation takes effect on the next keystroke rather than on the next rebuild, which is the only version of this control that is defensible in a review
- Indexes become permission-free artefacts, so they can be rebuilt, moved and shared across a workspace's developers without re-deriving an access list
- The degradation path is counted, so the cost of a slow authority appears as a quality number rather than as an unexplained acceptance dip

**What it costs**

- A third-party call sits inside the assembly budget, and its tail latency becomes this platform's tail latency
- The degradation to open-tabs context is invisible to the developer and visibly worse in acceptance, so a slow authority quietly degrades the product
- Negative-result caching is the only caching allowed, which forgoes most of the available saving

**Choose differently when.** If source hosting offered a reliable permission-change event, cached grants with push invalidation would be strictly better: the same correctness with none of the critical-path cost. For a single-tenant deployment where every developer can read every repository, the check is vacuous and should be skipped rather than optimised.

**Why it holds up over time.** "Authorise against the authority at the moment of access" is as old as access control and keeps being re-learned by every system that builds a derived index over protected data. Search platforms, data catalogues, vector stores and retrieval layers all face it, and the ones that cached the grant all eventually disclosed something. That will not change.

> **Lesson.** A derived index inherits its source's data and none of its permissions. Resolve access against the authority, every time, or accept that the index will eventually serve something it should not.

### Cost, capacity and the fleet

*Where the money goes in a workload whose majority output is discarded, and how many fleets it takes to hold the latency contract.*

#### ADR-13 · Cost is measured per accepted suggestion, and wasted generation is a capped, budgeted metric

**Status:** Accepted  ·  **Shown on views:** 15, 18

*More than half of what this fleet generates is never read. Which denominator makes that visible rather than invisible?*

**Context.** The natural unit for an inference service is cost per request, or cost per thousand tokens. Both are actively misleading here. At a 55% cancellation rate, cost per request averages the cheap discarded work with the expensive useful work and produces a number that improves when the fleet generates more rubbish faster. Cost per token is worse, because it rewards verbosity. Neither denominator lets anyone answer the question an engineering leader and a tenant administrator both actually ask: what did the spend buy. And because cancellation is driven by client behaviour the platform does not control (ADR-05), a denominator that hides cancellation also hides the single largest lever on fleet cost.

**Decision.** The headline cost metric is cost per accepted suggestion, with cost per displayed suggestion reported alongside it. Wasted generation — accelerator seconds spent on suggestions that were cancelled or never displayed — is a named, capped, budgeted metric with an owner and an alert, targeted at no more than 18% of accelerator seconds. Cost is attributed to tenant, language and model so that model routing (ADR-14) can be argued on cost per accepted rather than on cost per token. Per-tenant spend ceilings degrade along the published ladder (ADR-06) rather than cutting off.

**How it is realised on Azure.** Accelerator seconds are attributed at the batch scheduler per sequence and tagged with tenant, model digest, language and request shape, then joined in Azure Data Explorer against the interaction outcomes that say whether the suggestion was displayed and accepted. The same figures feed the admin API and the nightly export to the billing platform, so the number an administrator reads is the number they are charged on. Eager cancellation reclamation (50 ms at p99) is the primary control on wasted generation; admission refusal (ADR-04) is the second, since a refused request costs nothing at all.

| Option | Verdict | Reasoning |
|---|---|---|
| Cost per accepted suggestion, wasted generation capped | Chosen | Denominated in what the customer buys and exposes the dominant waste term; requires joining cost telemetry to outcome telemetry |
| Cost per request | Rejected | Hides the 55% that is discarded, and improves when the fleet wastes more work faster — an objective pointing the wrong way |
| Cost per thousand tokens | Rejected | The vendor's unit, not the customer's; rewards verbosity and is not comparable across model sizes or accelerator generations |
| Cost per retained suggestion (survival-weighted) | Deferred | The truest denominator, matching the quality metric in ADR-07, and it arrives a day late; worth adopting for monthly reporting once the retention pipeline is trusted |

**What it buys**

- The dominant waste term becomes a named metric with a cap and an owner instead of an emergent property of the cancellation rate
- Model choice can be argued on cost per accepted suggestion, which is what makes "use the smaller model" a measurable claim rather than a preference
- Administrators are shown and billed on the same figures, which removes the most common source of distrust in a usage-based product

**What it costs**

- The metric requires joining cost telemetry to outcome telemetry, so a telemetry gap becomes a billing and reporting gap
- Acceptance rate sits in the denominator, so a quality regression shows up as a cost regression and the two have to be disentangled before either is acted on
- A tenant whose developers are simply fast typists looks expensive per accepted suggestion through no fault of the platform

**Choose differently when.** If cancellation fell to near zero — a surface where suggestions are explicitly requested rather than fired on a keystroke — cost per request becomes honest again and the extra machinery is unnecessary. The decision is forced by the cancellation rate, so it should be revisited if the dominant surface changes.

**Why it holds up over time.** Cost per accepted suggestion will outlive the accelerator generation, the model size and the pricing model, because it is denominated in the thing the customer is buying. Cost per request, per token and per GPU-hour all stop being comparable the moment any of those three change, which they do every year.

> **Lesson.** Choose the denominator the customer would choose. A unit cost that improves when you waste more work faster is not a cost metric, it is a vanity metric with a currency sign.

#### ADR-14 · Two models routed by request shape — not one model, and not a deep cascade

**Status:** Accepted  ·  **Shown on views:** 08, 15, 16

*Single-line completions are the overwhelming majority and the cheapest; multi-line are rarer, more valuable and much more expensive. How many models serve that distribution?*

**Context.** One mid-size model is operationally simple and uniformly priced, and it overpays for the common case while underperforming on the rare one. A deep cascade — try the smallest, escalate on a confidence signal — is cheapest at the median and pays twice at the tail, because an escalated request has already spent the small model's latency before the large one starts, inside a budget that has no room for it. There is also a measurement problem that is specific to this product: the counterfactual suggestion is never generated, so a wrong routing decision produces no evidence. If the fast model was routed a request the large model would have completed better, nothing in the telemetry says so — the developer simply did not accept, which looks identical to the model having nothing useful to offer.

**Decision.** Two models, routed once at admission on features available before generation: request shape (single-line versus multi-line intent), language, file size and tenant tier. No mid-generation escalation, because escalation spends the budget twice. Routing is configuration rather than code, and routing quality is evaluated by holding a sample of traffic on forced routes so that the counterfactual exists for the sample even though it never exists for an individual request.

**How it is realised on Azure.** A fast single-line server and an escalated multi-line server run as separate AKS GPU node pools, each with its own affinity map and prefix cache, so a session's warm state belongs to one pool. The routing rule is versioned configuration in Azure SQL read through the policy cache, evaluated by the affinity router before admission so that placement and model choice are one decision. A small percentage of traffic is assigned to forced-route arms through the same experiment machinery as model promotion (ADR-09), which is what makes routing quality measurable at all.

| Option | Verdict | Reasoning |
|---|---|---|
| Two models, single routing decision at admission, forced-route sampling | Chosen | Keeps the common case cheap without spending the budget twice, and creates the counterfactual for a sample; costs two fleets to operate and two working sets to hold |
| One mid-size model for everything | Rejected | Simplest to operate and run; overpays on every single-line completion, which is most of 2.4 billion daily requests |
| Small-first cascade with mid-generation escalation | Rejected | Cheapest at the median and unaffordable at the tail: the escalated request has already spent part of a 300 ms budget before the larger model begins |
| Client-side routing from editor context | Deferred | Saves a server-side decision and puts a cost-bearing choice in an untrusted client that ships on its own cadence; revisit only with the server retaining a veto |

**What it buys**

- The majority case is served by the cheap model, which is the single largest structural saving available after prefix reuse
- Routing and placement are one decision, so a request is never routed to a model whose pool does not hold its warm prefix
- Forced-route sampling creates the only evidence that can show the routing rule is wrong

**What it costs**

- Two GPU pools means two working sets, two rollout surfaces and two sets of capacity decisions (ADR-15)
- A session that changes shape mid-edit may move pools and pay a cold prefix, so the routing feature set has to be stable within a session
- Routing quality is only measurable on the forced-route sample, so the rule is always evaluated on a small fraction of the evidence

**Choose differently when.** If one model could serve both shapes at the fast model's cost, two pools stop earning their complexity and should be collapsed immediately — this is a decision that should be re-tested at every model generation. Conversely, if a genuinely cheap confidence signal emerged before generation, a three-tier portfolio could beat two; the barrier is that the signal must be available at admission, not mid-stream.

**Why it holds up over time.** The durable part is the refusal to make a routing decision that cannot be evaluated: building the forced-route sample is what keeps the rule honest as the model portfolio changes. Routing by request shape rather than by mid-flight escalation is also a property of the latency budget rather than of the models, so it holds as long as the budget does.

> **Lesson.** Do not build a routing rule you cannot evaluate. If the counterfactual never exists, manufacture it for a sample — otherwise the rule becomes folklore that nobody can disprove.

#### ADR-15 · Each region is a complete working set; prefix caches are never replicated across regions

**Status:** Accepted  ·  **Shown on views:** 11, 17, 20

*Cross-continental round-trip time alone can consume most of a 300 ms first-token budget. How many fleets does that imply, and what is the cost of the answer?*

**Context.** A developer in Singapore served from Western Europe spends 150 to 200 milliseconds of the budget on network before a token is generated, which leaves nothing for assembly, admission and decoding. The latency contract therefore requires inference near the developer. But the accelerator fleet is the expensive, lumpy resource: splitting four thousand accelerators across three regions means three smaller fleets that batch less efficiently, three working sets rather than one, and three rollout surfaces for every model change. It also interacts badly with session affinity (ADR-01), because the warm-prefix working set is exactly the thing that fragments. Meanwhile geography pinning for data residency (ADR-10) forces some of the split regardless of latency, which means the question is not whether to split but whether to pretend the split is only about latency.

**Decision.** Each serving region is a complete working set: its own GPU pools, its own prefix caches, its own policy read replica and its own measurement path. Prefix caches are never replicated or transferred across regions — they are recomputable, so replicating them would couple regions for no durability gain. A region loss costs cold starts rather than correctness: sessions re-establish elsewhere with cold caches, inside a 5-minute RTO. The control plane has one write region with regional read replicas; the model registry is geo-replicated because weights are large, immutable and read everywhere.

**How it is realised on Azure.** Azure Front Door routes by geography and by tenant residency pinning rather than purely by latency, refusing rather than spilling when a tenant's declared geography is unavailable. Each region runs its own AKS GPU node pools, Container Apps request plane, and Redis affinity and policy caches; Azure Container Registry is geo-replicated so weights are pulled locally by digest. Azure SQL has one write region with read replicas per serving region, and Event Hubs plus Data Explorer are regional with aggregates consolidated asynchronously. Measurement is pinned to a separate zone within each region so a telemetry incident and a serving incident are not the same incident.

| Option | Verdict | Reasoning |
|---|---|---|
| Three complete regional working sets, no cache replication | Chosen | Holds the latency contract and keeps a region loss to cold starts; pays with fleet fragmentation, lower batch efficiency and three rollout surfaces |
| One global fleet behind a global edge | Rejected | Best batch efficiency and one working set, and it cannot meet a 300 ms p95 for anyone more than a continent away |
| Regional request planes, centralised inference | Rejected | Looks like a compromise and keeps the whole network cost on the critical path, which is the part of the budget that cannot be optimised |
| Cross-region prefix cache transfer on failover | Rejected | Couples regions and adds a transfer cost for state that can simply be recomputed — the wrong trade for a cache that is not a system of record |

**What it buys**

- The latency contract is attainable for developers on three continents rather than one, which is the difference between a product and a demo
- A region loss is a cold-start event with a 5-minute RTO, because nothing in the region held authority over anything
- Data residency and latency are served by the same topology, so pinning is not an exception to the architecture

**What it costs**

- Three smaller fleets batch less efficiently than one large one, which is a permanent utilisation cost
- The warm-prefix working set fragments, so each region needs its own headroom and the sum exceeds what one fleet would need
- Every model promotion, rollback and configuration change has three surfaces, which multiplies the rollout machinery and the ways a ramp can be inconsistent

**Choose differently when.** If network latency to a central region fell below roughly 30 milliseconds for the served population — a single-country deployment, or a future where that is simply true — one fleet becomes better on every axis and the regions should be collapsed. The sovereign single-tenant deployment in phase 3 is the opposite case: a region of one tenant, where fragmentation is the whole point and efficiency is explicitly not the objective.

**Why it holds up over time.** "Replicate authority, recompute derived state" is the transferable rule, and it is what keeps a multi-region design simple: the only things that cross a region boundary are the small immutable ones. That will hold through any change in accelerator, model or cloud. What will change is the number of regions, which is a function of network physics and of where the customers are.

> **Lesson.** Replicate what is authoritative and recompute what is derived. A cache that crosses a region boundary has bought you a coupling and sold you nothing.

## Every package used, in one table

These terms are used precisely in this package. Several are used loosely in the wider literature on inference serving, and the difference matters when reading the decision records.

| Package | What it is | What it does here | Considered instead |
|---|---|---|---|
| Warm session | An editing session whose key-value prefix state is resident on a particular accelerator replica, so the next keystroke's prompt is almost entirely already computed. | The platform's unit of capacity: fleets are planned, scaled and alerted on in warm sessions rather than in requests per second. | "Connection" or "request", neither of which carries the state that makes the economics work. |
| Prefix cache | The retained per-sequence state for the stable leading segment of a prompt, tiered across device memory, host memory and node-local disk, and partitioned by tenant. | Simultaneously the largest latency win and the largest cost win in the system, and therefore the thing the whole architecture is arranged around. | "KV cache", used interchangeably here; "response cache", which is a completely different and much weaker mechanism. |
| Fill-in-the-middle | A request shape carrying the text before and after the cursor, rather than a prefix alone, so the model completes into a context that continues after the insertion point. | The default and only request shape; prefix-only continuation is not offered, because it mis-models what a developer is doing in the middle of a file. | "Completion", which in most of the literature means prefix-only continuation. |
| Displayable | The point in a stream at which the suggestion is syntactically complete enough to render as ghost text, which is earlier than generation finishing. | What the latency contract is measured against: p95 ≤ 300 ms to displayable, not to the last token. | "Time to first token", which is close but counts a token the client cannot yet paint. |
| Deadline | The client's remaining time budget in milliseconds, stamped at the edge and propagated with the request, after which no result may be returned. | The input to admission control, and the reason a late suggestion is never delivered even when it is ready. | "Timeout", which usually means the server gives up rather than the server refuses to start. |
| Wasted generation | Accelerator seconds spent producing suggestions that were cancelled before display or never displayed. | A named, capped, owned metric targeted below 18% of accelerator seconds, because it is the dominant cost term in this workload. | Usually unnamed, and therefore unbudgeted, in inference-serving designs. |
| Accepted-character retention | The proportion of an accepted suggestion's characters still present in the file after a declared interval — 120 seconds, and again at 24 hours. | The canonical quality metric, because acceptance alone rewards a suggestion for being plausible rather than for being right. | "Acceptance rate", which is a fast companion signal here and the primary metric almost everywhere else. |
| Control arm | A stable, randomly assigned fraction of developers permanently served the previous known-good model and prompt recipe. | The only instrument that distinguishes a worse model from a changed population, for the one failure class that produces no error signal. | A per-experiment control, which leaves the system blind between experiments. |
| Zero retention | A tenant mode in which no prompt, suggestion or code-bearing telemetry is persisted anywhere, implemented as the absence of a configured sink rather than as a filtered write. | The enterprise default, and the claim the whole adoption argument rests on. | "Retention disabled", which describes a setting rather than a system. |
| Affinity loss | The eviction of a live session's prefix state because its replica was drained, preempted or scaled in. | A measured latency penalty with its own counter, and explicitly not an error — a design where it errored would make session affinity unacceptable. | Usually reported as a pod restart or a connection reset, which hides the cost where nobody is looking for it. |