Architecture Views

Between two keystrokes the prompt barely changes. Does the fleet exploit that, and if it does, what is given up to make the exploitation possible?

Context

A completion request's prompt is thousands of tokens of surrounding code. The next request in the same session, sixty milliseconds later, is the same thousands of tokens plus one character. Recomputing attention over the whole prefix on every keystroke is affordable at a demo's traffic and ruinous at 2.4 billion requests a day: it is simultaneously the largest latency cost and the largest money cost in the system. Reusing the cached prefix state is therefore not an optimisation but the economic basis of the product. The difficulty is that the cached state lives in a particular replica's memory, which means the request must go where the state is. That one requirement breaks the assumption every other tier in a modern platform is built on — that any replica can serve any request — and it does so in the tier that is hardest to operate: the one holding four thousand accelerators.

Decision

Inference is session-affine. A session identifier, stable for the editing session, is routed to the replica that holds that session's key-value prefix; capacity is planned, scaled and alerted on in units of concurrent warm sessions rather than requests per second. Every other tier — edge, context assembly, post-processing, telemetry ingest — is stateless and disposable. When no replica holds the session, any replica may take it and pay the cold-prefix cost; affinity is a preference with a measured penalty, never a queueing constraint.

How it is realised on Azure

AKS GPU node pools run the model servers, with the affinity map held in Azure Cache for Redis as (tenant, session) → replica with a session-length TTL. The affinity router runs in the stateless request plane on Container Apps and resolves placement before admission. Horizontal pod autoscaling is driven by custom metrics — admission rejection rate and queue-wait at p99 — published to Azure Monitor, not by GPU utilisation. Node pools use a drain protocol that stops accepting new sessions while completing in-flight sequences, with the eviction surfaced as an affinity-loss counter rather than as a pod restart.

Options weighed

• ChosenSession-affine placement with a measured cold-start fallback: Buys an 85% prefix hit rate and the latency and cost that depend on it; pays with placement logic, a drain protocol, and scale-in that evicts live sessions
• RejectedWholly stateless inference, no prefix reuse: Trivial to scale and to drain, and multiplies accelerator spend several times over while putting a full prefix recompute inside a 300 ms budget on every keystroke
• DeferredStateless inference over a shared remote prefix cache: Decouples routing from state, and pays a cache transfer on the critical path plus a store every request must touch; revisit if cross-node interconnect makes the transfer cheaper than the recompute
• Right elsewhereClient-side caching of generated continuations: Right for a product with a small, repetitive completion space — a SQL console, a form DSL — and useless where the prefix is the developer's own unique file

Consequences

What it buys

• An 85% prefix hit rate makes p50 first-token latency 120 ms rather than a full recompute, and takes the dominant term out of the cost model at the same time
• Capacity planning becomes honest: the question "how many developers can this fleet hold" has an answer in warm sessions, which is a quantity operators can reason about
• Cancellation becomes cheap. Evicting a sequence while keeping its prefix warm means the 55% of requests nobody reads cost a batch slot rather than a recompute

What it costs

• Scale-in is correctness-adjacent: removing a replica evicts live sessions, so autoscaling must be asymmetric and drains must be graceful rather than terminated
• The inference tier needs placement logic, an affinity store, and a failure mode — affinity loss — that no other tier has
• Multi-region stops being a latency decision and becomes a working-set fragmentation decision (ADR-15)

Choose differently when

If the prompt were not incremental — a product where each request's context is unrelated to the last, such as a batch code-explanation service or a one-shot refactoring tool — there would be no warm prefix to route to, and a stateless pool would be strictly better. The decision is justified by the keystroke cadence, and it should be revisited the moment the dominant surface stops being per-keystroke. It would also flip if an interconnect made transferring a prefix cheaper than recomputing it: at that point the shared remote cache becomes the better design, keeping the reuse and giving back painless drains.

Why it holds up over time

"Place the request where the expensive state already is" does not depend on today's attention implementation. Any generative decoder keeps per-sequence state, and any system serving an incremental high-frequency editing signal will find the previous request's state is almost all of the next one's. The mechanism may become a compiled state, a learned summary or something unnamed; the decision that clients are placed rather than balanced survives the change. What would not survive is making the cached state authoritative — the moment a prefix cannot be recomputed, affinity loss becomes an error instead of a latency penalty, and the whole design stops degrading gracefully.

Device memory is the scarcest resource on the fleet and it has to hold both the batch and the cached prefixes. How is it divided, and what happens to a prefix that is evicted?

Context

A replica's accelerator memory is contended three ways: model weights, in-flight batch activations, and cached prefixes. Give more to prefixes and the batch narrows, which lowers throughput; give more to the batch and the hit rate falls, which raises latency. Meanwhile the obvious efficiency win is tempting and dangerous: prefixes are mostly code, much of it shared — the same framework imports, the same license header, the same generated client. A cache keyed purely on content would deduplicate across tenants and raise the hit rate materially. It would also mean one tenant's cached state can be reached through another tenant's request, which converts the fleet's chief efficiency mechanism into its chief isolation risk, in a product whose input is the customer's source code.

Decision

The prefix cache is three tiers — accelerator device memory, host DRAM, then node-local NVMe — with eviction cascading downward rather than discarding. The cache key includes the tenant identifier as a hard partition: no cache entry is ever reachable from a request belonging to another tenant, regardless of content equality. Cross-tenant deduplication is refused as a design, not configured off. Device-memory allocation between batch width and prefix residency is a tuned, published parameter with the hit rate and batch occupancy both reported.

How it is realised on Azure

Model servers on AKS GPU node pools manage the device tier in-process and spill to host DRAM, then to the node's local NVMe temp disk, which is sized for the working set rather than for logs. Tenant partitioning is a prefix on the cache key, enforced in the server rather than by convention. Tier hit rates (HBM / DRAM / NVMe) and batch occupancy are exported to Azure Monitor as first-class capacity signals, and the device-memory split is a per-model-pool configuration value in the registry's decoding config rather than a deployment flag.

Options weighed

• ChosenThree tiers, tenant-partitioned, no cross-tenant deduplication: Keeps a re-placed or evicted session on a local disk read rather than a full recompute, and makes the isolation claim structural; gives up a real hit-rate gain
• RejectedContent-addressed cache shared across tenants: The largest available efficiency win and an unacceptable one: a cross-tenant read path over customer source code, defended only by the correctness of a cache key
• RejectedDevice memory only, discard on eviction: Simplest, and makes every eviction a full recompute — which turns ordinary batch pressure into a visible latency incident
• DeferredRemote shared cache tier behind NVMe: Would let a drained session recover its prefix on a different node; deferred until the transfer cost is reliably below the recompute cost at p95

Consequences

What it buys

• Eviction degrades instead of resetting: a prefix pushed out of device memory is a disk read, not a recompute, which is what keeps the cold-prefix penalty at 180 ms rather than several hundred
• The tenant isolation claim is enforced by the key rather than by a policy, which is the form a security reviewer can actually accept
• The batch-versus-residency split becomes a visible, tunable number with two reported metrics, instead of an emergent property nobody owns

What it costs

• A measurable hit-rate gain is given up permanently, and with it some accelerator spend
• Node-local NVMe becomes a sizing requirement on the GPU node pool, which constrains instance selection
• Three tiers means three hit rates to reason about, and a regression in the lowest tier is easy to miss because it only shows up at the tail

Choose differently when

In a single-tenant deployment — a sovereign or on-premises installation for one organisation, which is phase 3 of the requirement — the partition buys nothing and content-addressed deduplication becomes straightforwardly correct. The decision is a multi-tenancy decision, not a caching one. It would also change if device memory stopped being the binding constraint: with an order of magnitude more of it, the tiering below device memory would matter much less and could be dropped for simplicity.

Why it holds up over time

"Efficiency mechanisms that cross a trust boundary are not efficiency mechanisms" is the durable half of this record. The specific tiers will be replaced — memory hierarchies change every accelerator generation, and the names HBM, DRAM and NVMe will not last — but the rule that a cache key carries the tenant, and that sharing is refused rather than configured, will still be the right call when the hierarchy has four levels and different names.

Prompt composition is usually treated as a quality question. What happens to the cache when it is treated only as a quality question?

Context

Context assembly has freedom in what it includes and in what order. Treated as a relevance problem, the natural instinct is to put the most relevant material closest to the cursor — retrieved snippets, the most recently edited region, whatever the ranker scored highest. All of those change between keystrokes. A prefix cache is valid only up to the first token that differs, so a single volatile item placed early invalidates the entire cached prefix on every keystroke, and the 85% hit rate the whole architecture depends on collapses to near zero. The failure is invisible in testing: quality metrics look fine, latency looks fine at low traffic, and the cost and tail latency only appear at scale. This makes prompt ordering an architectural constraint masquerading as an authoring preference.

Decision

The prompt has a declared two-part structure: a stable leading segment — the file's identity, the language preamble, the imported-symbol block, the retrieved repository snippets selected for the session rather than for the keystroke — followed by a volatile tail holding the cursor neighbourhood. Sources are assigned a priority that determines both truncation order and position, and the lowest-priority source is dropped first with the drop recorded. The ordering rule is versioned as a prompt recipe, and the recipe version is logged against every request so a quality change is attributable to composition rather than to the model.

How it is realised on Azure

The prompt assembler runs in the stateless request plane on Container Apps; the recipe is versioned configuration in the Azure SQL control plane, read through the same bounded-staleness Redis cache as policy, so a recipe change takes effect within 30 seconds without a deployment. Retrieved snippets are selected once per session and cached with the session rather than re-ranked per keystroke. The recipe version is a column on suggestion_request in the telemetry model; prefix stability is exported as its own metric so that a recipe change that hurts the cache is visible within minutes rather than in next month's bill.

Options weighed

• ChosenDeclared stable segment plus volatile tail, versioned recipe: Protects the hit rate and makes quality attributable; constrains what the ranker may do and when it may do it
• RejectedRelevance-ordered prompt, most relevant nearest the cursor: The intuitive design, and it invalidates the cache on every keystroke — a quality decision that silently spends the latency budget it was supposed to help
• DeferredPer-keystroke re-ranking with cache-aware reordering: Could get both if the reorderer could prove stability; deferred because it adds a correctness obligation to a component on the critical path
• Right elsewhereFixed prompt template with no retrieval at all: Right for a narrow DSL or a single-file scripting surface, where open tabs and repository context add nothing worth a token

Consequences

What it buys

• The hit rate the architecture depends on is protected by a rule rather than by hoping, and prefix stability has a metric that fails loudly
• A quality regression can be traced to a recipe version without retaining a single prompt, which keeps attribution compatible with zero retention
• Retrieval becomes a per-session cost rather than a per-keystroke one, which also removes it from the critical path at the frequency that matters

What it costs

• The ranker cannot act on the most recent keystroke, so some genuinely better context is excluded by position
• Per-session snippet selection goes stale within a long editing session, and refreshing it is a deliberate cache-invalidating event that has to be scheduled
• Prompt authoring becomes a constrained activity with a review step, which slows down the experiments that quality work depends on

Choose differently when

If prefix reuse were abandoned (see the flip on ADR-01), this constraint evaporates and prompt order becomes purely a relevance question. Short of that, it would loosen if models became robust to context position in a way that let the volatile material sit anywhere — but that is a claim about relevance, not about caching, and the cache would still prefer the stable ordering.

Why it holds up over time

The transferable property is that a cache's hit rate is a function of the request's construction, not of the cache's implementation — so whoever composes the request owns the hit rate whether they know it or not. That will be true of any prefix-reusing system. Making the composition a versioned, measured artefact rather than code is what keeps it true as the team that writes prompts changes.

When the fleet cannot serve a request inside its deadline, what does the developer get?

Context

The usual answer in a serving system is to queue: accept the work, do it as fast as possible, and let latency degrade under load. That answer is wrong here, because the value of a suggestion falls to zero and then goes negative. A suggestion that arrives after the developer has typed past it is not a slow success — it is grey text appearing over a line they have already written, which costs attention and offers nothing. Worse, under burst the queueing design spends accelerator seconds on suggestions nobody will read, so the fleet is most wasteful exactly when it is most contended. The 55% baseline cancellation rate means this is not an edge case: the system's normal state is that most work in flight is already doomed.

Decision

Every request carries the client's remaining deadline. An admission gate compares that deadline against current queue depth and estimated generation time and refuses immediately — before any accelerator work begins — anything it predicts will miss. A refused request returns an explicit empty result, which the client treats as "the model had nothing to say". Nothing is queued without bound, and no result is returned after its deadline has passed, even if it is ready.

How it is realised on Azure

The deadline is stamped at the Front Door edge after identity and quota, propagated as a request header through the Container Apps request plane, and evaluated by the admission gate immediately before placement. The gate's inputs are the replica's in-flight sequence count, measured queue-wait at p99 from Azure Monitor, and a per-model generation-rate estimate from the registry's decoding config. Refusals are a first-class metric and the primary autoscaling signal for the GPU node pools (ADR-01); the client translates an empty result into no ghost text and widens its own debounce on repeated refusals.

Options weighed

• ChosenDeadline-aware admission, refuse at the door: Zero accelerator cost for work that cannot land, and a load signal that is countable; requires predicting generation time before generating
• RejectedQueue and serve as fast as possible: The default in most serving stacks, and here it spends the most accelerator seconds on the least valuable work precisely under burst
• RejectedClient-side abandonment only, server finishes the work: Simplest contract and strictly worse economics: the server cannot tell a cancelled request from a live one and pays for both
• DeferredReturn the best partial result at the deadline: Attractive for multi-line suggestions and risky: a truncated multi-line completion can be more misleading than none; revisit once displayability can be judged mid-stream

Consequences

What it buys

• A 4× burst becomes a capacity question with a number attached — the refusal rate — rather than a latency incident with a shrug attached
• Accelerator seconds are not spent on suggestions that cannot be displayed, which is what makes the cost-per-accepted target reachable
• The refusal rate is a leading autoscaling and alerting signal, which average GPU utilisation can never be

What it costs

• Generation time must be estimated before generation, from features available at admission; a bad estimator either refuses work it could have served or admits work it cannot
• Developers in a contended region experience the product as intermittently absent, which is a silent degradation they cannot report usefully
• The empty result is indistinguishable from "the model genuinely had nothing", so the refusal rate has to be instrumented server-side or the degradation is invisible

Choose differently when

If the product's value did not decay inside a second — a batch completion pass over a file, a pull-request suggestion run, an agentic edit the developer waits for — queueing would be correct and refusing would be perverse. The decision belongs to the keystroke surface specifically. It would also soften if accelerator capacity became cheap enough that peak could simply be provisioned for, at which point the gate still exists but almost never fires.

Why it holds up over time

Deadline propagation plus refuse-rather-than-queue is a protocol property, not a framework feature, and it is right wherever the value of a response decays faster than the queue drains. It has outlasted several generations of serving infrastructure already. The part that would date is the estimator, which depends on the decoder; the part that will not is making the refusal countable, because that is what lets capacity be planned honestly.

The editor is the one component the platform cannot control and the only one that knows the keystroke. How much of the latency contract is allowed to depend on it?

Context

Three of the mechanisms that make this system affordable can only exist in the editor. Debounce requires knowing that a keystroke happened and that no further keystroke followed for sixty milliseconds. Cancellation requires knowing that the developer kept typing. Suppression inside string literals, comments and already-complete lines requires the editor's own parse of the buffer. None of these can be done server-side at this budget, because the information arrives with the keystroke and the round trip is the thing being avoided. But the editor plugin is also the least trustworthy component in the system: it runs on the developer's machine, it ships on its own release cadence, and a single bad version can change fleet economics by tens of percent with nothing server-side having changed.

Decision

The client owns debounce, single-in-flight-per-session, cancel-on-keystroke and local suppression, and these are published as a versioned client contract with a supported version floor. No policy decision, filter, quota or authorisation is enforced only on the client: every one of them is independently enforced server-side, so a modified or defective client can waste capacity and cannot gain access or bypass a filter. Clients identifying a version below the floor are refused rather than served degraded, and a server-side kill switch can force any client version into a reduced mode without a client release.

How it is realised on Azure

The contract is versioned alongside the Suggestion API and enforced at the Front Door edge, which reads the client version header and consults the flag service before anything else. The version floor, the reduced-mode flag and the per-tenant and per-developer kill switches live in the Azure SQL control plane and reach the edge through the policy cache, taking effect for new requests within 30 seconds. Per-seat rate limiting at the edge bounds what a misbehaving client can cost; cancellation-rate distribution is reported per client version so a bad release is visible as a distribution shift rather than as an unexplained spend increase.

Options weighed

• ChosenClient owns the timing mechanisms; server enforces everything that matters: The only arrangement that fits the budget and survives a hostile client; accepts that capacity cost is partly outside the platform's control
• RejectedServer-side debounce and suppression: Requires the round trip that debounce exists to avoid, and still cannot see the keystroke that should have cancelled the request
• RejectedTrust the client for quota and suppression as well: Cheaper and indefensible: a plugin is a program on someone else's machine, and a filter it can skip is not a filter
• DeferredA thin local daemon owned by the platform, with the editor as a shell: Gives the platform a controlled client without the round trip; deferred because it multiplies the install and update surface across four editors and three operating systems

Consequences

What it buys

• The timing mechanisms sit where the information is, which is what makes the 300 ms budget attainable at all
• Authorisation, policy and filtering remain enforceable against a client the platform does not control, so adoption does not rest on client integrity
• The version floor and reduced mode give an operator two blunt but immediate levers when a release goes wrong

What it costs

• Accelerator spend is partly determined by client behaviour the platform can only observe after the fact
• Version skew is permanent across four editors, so the contract must stay backward-compatible for longer than is comfortable
• Every policy must be implemented twice — once in the client for latency, once on the server for truth — and the two can drift

Choose differently when

If the platform shipped the editor itself, the client would stop being untrusted and much of the duplication could go. That is the situation for a vendor whose own IDE is the only surface, and their design would legitimately differ. It would also flip if a server round trip got cheap enough — on a local network, or for an in-browser IDE served from the same region — that debounce could move server-side.

Why it holds up over time

"Put the mechanism where the information is; put the enforcement where the trust is" is the durable statement, and it will survive every editor, protocol and plugin API in this space. What dates is the specific list of mechanisms: a future client may own more of the pipeline, or less, but the split between latency-driven placement and trust-driven enforcement stays.

When there is not enough capacity for everything, who loses their suggestion first — and is that a decision or an accident?

Context

Admission control (ADR-04) decides whether a single request can make its deadline. It does not decide which requests should be sacrificed when the fleet as a whole is short. Without an explicit answer, the answer emerges from queue mechanics: whoever happens to arrive during the spike loses, which means the loss is distributed by luck and is impossible to explain to a tenant afterwards. Worse, the most expensive work — multi-line suggestions, speculative prefetch — survives at the same rate as the cheapest, so shedding recovers far less capacity than it costs in goodwill. A fleet serving 1.2 million developers with a 5× trough-to-peak daily swing will be short regularly enough that this needs to be a design rather than an emergent property.

Decision

Shedding is ordered and published: speculative prefetch is shed first, then multi-line suggestions, then lower service tiers, and single-line user-initiated requests on the highest tier are shed last. Shedding is always a refusal — an explicit empty result — and never an extension of latency. A tenant that has reached a spend ceiling degrades along the same declared ladder (smaller model, single-line only, then suspension) rather than being cut off without warning.

How it is realised on Azure

The shedding order is configuration in the Azure SQL control plane, read through the policy cache and evaluated by the admission gate alongside the deadline check. Request shape (single-line versus multi-line) is already known at admission because it determines model routing (ADR-14), so no extra classification step is needed. Shed counts are exported per class and per tenant to Azure Monitor, and the per-tenant degradation ladder is surfaced in the admin API so an administrator sees which rung they are on before they see the bill.

Options weighed

• ChosenPublished shedding order, refusal only: Recovers the most capacity per unit of lost value and makes degradation explainable; requires classifying request value at admission
• RejectedFair-share shedding across tenants: Equitable and inefficient: it protects cheap and expensive requests equally, so it sheds more requests to recover the same capacity
• RejectedLatency degradation under load: The implicit default, and the one outcome this product cannot absorb — a late suggestion is worse than no suggestion
• Right elsewherePriority bidding per tenant: Right where customers genuinely value latency differently and will pay for it; here it would make the product's core experience a purchasable variable, which is a commercial decision rather than an architectural one

Consequences

What it buys

• Degradation becomes a decision with an owner and a published order, which is the difference between a managed shortage and an incident
• Shedding the expensive classes first recovers more accelerator capacity per refused request, so the shortage ends sooner
• Spend-ceiling behaviour and capacity-shortage behaviour share one ladder, so there is one degradation story rather than two

What it costs

• Multi-line suggestions are structurally the least reliable feature of the product, which is a permanent asymmetry users will notice
• Request-class value is a judgement encoded in configuration, and a wrong ordering is only discoverable from acceptance data afterwards
• Tier-based shedding couples the architecture to the commercial packaging, which then becomes hard to change

Choose differently when

If capacity were elastic on the timescale of a burst — accelerators available in seconds rather than minutes — shedding would matter far less and provisioning would do the work instead. It would also change if multi-line suggestions turned out to carry most of the measured retention, in which case the ordering is simply wrong and the data should overturn it.

Why it holds up over time

"Shed by value, in an order you published before the incident" outlives every scheduler. The specific classes here are tied to this product's shapes, but the principle — that a shortage should be resolved by a decision made in advance rather than by queue mechanics in the moment — is one of the few operational rules that has never stopped being true.

Nobody tells this system whether a suggestion was good. What stands in for that, and what does the substitute get wrong?

Context

The available signals are all behavioural. A suggestion is displayed, or cancelled before display; if displayed it is accepted or ignored; if accepted it survives or is deleted. Acceptance rate is the obvious metric: it is immediate, abundant and easy to compute. It is also systematically misleading in the one direction that matters. A model that is more confident and more fluent produces suggestions that look right and get taken, and some fraction of those are deleted twenty minutes later when the developer discovers the API does not exist. Optimising acceptance therefore optimises plausibility, which is precisely the failure mode of the underlying technology. Retention — how much of an accepted suggestion still exists in the file after a delay — sees the difference, and arrives a day late.

Decision

Accepted-character retention is the canonical quality metric, measured at 120 seconds and again at 24 hours after acceptance. Acceptance rate and time-to-first-token are fast guard rails that may halt a rollout on their own but may never promote one. A model that improves acceptance and regresses retention is refused. Both metrics are always reported against a control arm (ADR-09) rather than against the model's own history.

How it is realised on Azure

The editor plugin probes the accepted span at 120 seconds and reports surviving characters as an asynchronous telemetry event to Event Hubs; the 24-hour measurement is derived from a later session's buffer state for the same span. Events land in Azure Data Explorer, where retention and acceptance are materialised per model digest, language, prompt recipe version and experiment arm. The promotion gate reads those aggregates; thresholds live in the control plane so a gate can be tightened without a deployment. Only character counts and span offsets are transmitted — never the characters themselves — which is what keeps the metric compatible with zero retention.

Options weighed

• ChosenRetention canonical, acceptance as a guard rail: Measures usefulness rather than plausibility; costs a day of latency in the decision loop and a client-side probe
• RejectedAcceptance rate as the primary metric: Immediate, abundant, and rewards exactly the failure mode of a confident model; this is the industry default and it is the wrong default
• RejectedExplicit developer feedback (thumbs up/down): Sparse, biased toward the annoyed, and an interruption in a surface whose entire value proposition is not interrupting
• DeferredDownstream signals — test pass rate, review comments, revert rate: The truest measure of usefulness and far too slow and confounded to gate a weekly model cadence; worth building as a quarterly check

Consequences

What it buys

• The metric that gates promotion measures whether the suggestion was useful rather than whether it was convincing, which is the only defence against optimising for plausibility
• Retention is computable from character counts and offsets alone, so quality measurement survives a zero-retention tenant
• Per-language and per-recipe retention makes it possible to see a model that improved the mean and ruined one language

What it costs

• A one-day decision latency, which bounds how fast models can be iterated and forces a capped canary in the meantime
• The probe depends on the client, so a plugin release can move the headline quality metric without the model changing
• A span that was accepted and then legitimately refactored counts as lost retention, which puts a noise floor under the metric

Choose differently when

If the product moved from inline completion to whole-function or agentic edits, the unit of measurement changes — a diff accepted and then reworked is normal rather than a failure — and retention at the character level stops being meaningful. At that point test outcomes and review signals become the right instrument despite their latency.

Why it holds up over time

Every offline benchmark eventually saturates or gets trained against. What does not saturate is whether a human kept the output. The question retention asks — did this survive contact with the work — is independent of what the model can do, so it will still be the right instrument when the model writes whole modules. The 120-second probe is an implementation detail; the choice of a survival metric over an uptake metric is not.

An offline suite over held-out completion contexts is cheap and fast. Why is it not allowed to decide anything?

Context

Offline evaluation is attractive because it is fast, repeatable and available before any developer is exposed. The problem is that its correlation with retention is weak and unstable: exact-match and edit-distance scores reward a model that reproduces the held-out continuation, which is not the same as producing something the developer keeps, and the correlation shifts whenever the corpus, the languages or the model family changes. Treating such a score as a verdict means putting a model in front of 1.2 million developers on the strength of a proxy whose relationship to the real metric nobody can state. Treating it as nothing, however, wastes the one check that can catch an obviously broken candidate for a few dollars.

Decision

The offline suite is a gate with one direction of authority: it can refuse a candidate before it reaches a canary, and it can never promote one. A model that has not been evaluated cannot be promoted at all. Promotion requires an online retention verdict (ADR-07) measured against the control arm. Offline thresholds are set to catch gross failures — wrong language, degenerate output, collapse on a common construct — rather than to rank near-equivalent candidates.

How it is realised on Azure

The harness runs as a job against a held-out corpus built only from licensed material, stored in Blob Storage and versioned alongside the registry. A model version in Azure Container Registry carries an eval_run_id, and the Model API refuses a traffic split for any digest without one. Thresholds are control-plane configuration, so tightening the gate is a configuration change and not a pipeline release. Results are retained for two years so that a later retention surprise can be checked against what the offline suite said at the time — which is how the correlation gets re-estimated rather than assumed.

Options weighed

• ChosenOffline as a refusing-only gate, retention as the verdict: Catches broken candidates cheaply without granting authority to a weak proxy; costs a day of canary before anything can ship
• RejectedOffline score as the promotion criterion: Fast, cheap, and the most common way a completion product regresses — it promotes the model that best reproduces the corpus, not the one developers keep
• RejectedNo offline evaluation; canary everything: Exposes developers to candidates a mechanical check would have caught, and burns canary capacity that is the scarce resource in the loop
• Right elsewhereHuman evaluation panels on sampled completions: Right for a lower-volume, higher-stakes surface; at a weekly cadence across dozens of languages the panel becomes the bottleneck and its judgement does not generalise

Consequences

What it buys

• An obviously broken candidate is stopped for the price of a batch job, before any developer is exposed
• Nothing is promoted on a metric whose relationship to usefulness cannot be stated, which is the whole point of the asymmetry
• Keeping offline results alongside later retention outcomes turns the correlation into something measured rather than assumed

What it costs

• Every candidate pays a canary day, even one that is obviously better offline
• Near-equivalent candidates cannot be ranked before exposure, so canary capacity rather than compute becomes the limit on iteration speed
• A licensed-only corpus is narrower than production traffic, so the gate is blind to constructs that only appear in customers' code

Choose differently when

If an offline metric were ever shown to predict retention reliably across languages and model families, the asymmetry could be relaxed and promotion could move offline — which would transform iteration speed. That is a measurement result, and the retained eval history exists precisely so it could one day be demonstrated rather than claimed.

Why it holds up over time

The durable statement is about the direction of authority, not about the metric: a cheap proxy may veto and may not decide. That rule holds for every proxy in every system, and it is what prevents proxy metrics from quietly becoming objectives. It will survive the replacement of exact-match scoring by whatever succeeds it.

Which failure in this system produces no error, no alert and no page — and what instrument detects it?

Context

Every other failure class here announces itself. Latency misses show up in a percentile, saturation in admission refusals, dependency failure in a degradation counter, filter failure in a suppression count. Quality is different: a model that has become less useful looks exactly like a healthy fleet. Latency is fine, errors are zero, acceptance drifts by a percent and nobody notices for a month, by which time it is attributed to a tooling change or a seasonal shift. Comparing against the model's own history does not help, because the population drifts too — new tenants, new languages, a change in the mix of work. The only reliable reference is a concurrent population served by a known-good configuration.

Decision

A control arm is held back permanently, not per experiment: a stable, randomly assigned fraction of developers continue on the previous known-good model and prompt recipe. All quality figures are reported as differences against that arm. The gate that halts a rollout reads the difference, not the absolute. Measurement failures — a telemetry gap, a broken probe, an arm whose assignment has drifted — are alerted separately from quality regressions, so an on-call engineer can tell a broken instrument from a worse model.

How it is realised on Azure

Arm assignment is stable per developer and held in the Azure SQL control plane as experiment_arm rows with an is_control flag, so an assignment survives restarts, plugin upgrades and cache flushes. Aggregates in Azure Data Explorer are computed per arm and per language. The control arm runs on its own share of accelerator capacity with the previous model mounted, which is capacity permanently unavailable to the current one. Arm-balance and telemetry-coverage checks run continuously and page a different rotation from quality alerts.

Options weighed

• ChosenPermanent held-back control arm, all figures as differences: The only instrument that distinguishes a worse model from a changed population; costs permanent capacity and leaves some developers on an older model
• RejectedPer-experiment control, no standing arm: Cheaper and blind between experiments, which is exactly when a slow regression accumulates unnoticed
• RejectedCompare against the model's own history: Confounded by population drift, seasonality and plugin releases — it answers a different question from the one being asked
• DeferredSynthetic canary traffic against a fixed probe set: Useful for serving health and useless for quality, because a synthetic developer never accepts or deletes anything; worth keeping as an availability probe only

Consequences

What it buys

• The one failure class with no error signal acquires an instrument, and an automated rollback can act on it without a human noticing first
• Population drift stops being confounded with model quality, which makes a 1.5% effect claim defensible
• Separating measurement-failure alerts from quality alerts removes the worst minute in the system: not knowing whether the fleet or the thermometer is broken

What it costs

• A permanent slice of developers gets a deliberately older model, which is a real and ongoing product cost
• Control-arm capacity holds the previous model warm forever, taking working set from the current one (ADR-15)
• Permanent assignment creates a long-lived population that must be rebalanced carefully, because a drifted arm silently invalidates every figure computed against it

Choose differently when

If quality could be measured absolutely — a reliable automated correctness check on each suggestion — the control arm's main job would disappear and it could shrink to an availability probe. Short of that, the arm shrinks only if the developer population stops drifting, which for a growing product it will not.

Why it holds up over time

"A metric with no error signal needs a concurrent reference, not a historical one" is the transferable half, and it applies far beyond this product — to recommendation quality, ranking relevance and anything else judged by behaviour. The cost of the arm will fall as capacity gets cheaper; the need for it will not change at all.

An enterprise security reviewer is told retention is disabled. Why should they believe it, and what would make the claim checkable?

Context

The input to this system is the customer's source code, which is usually their most valuable asset and often the one with the strictest contractual controls. A reviewer evaluating adoption knows that a setting can be misconfigured, re-enabled by a deployment, bypassed by a new code path, or quietly contradicted by infrastructure nobody considered — crash dumps, request logs, distributed traces, a debugging flag left on in one region. They are right to discount a configuration claim, because the history of such claims is bad. Meanwhile the platform genuinely needs telemetry: without interaction data there is no quality measurement at all (ADR-07, ADR-09), and without quality measurement the product cannot improve.

Decision

Telemetry is split at the boundary into a code-bearing stream and a non-code stream, and for a zero-retention tenant the code-bearing path does not exist: no sink is configured, no buffer is allocated, and the scrub is enforced at the sink as well as at the source so that infrastructure which does not know about retention mode cannot create one. Zero retention is the default for enterprise tenants. Diagnostic retention is opt-in, bounded at 24 hours, and drawn in red on the data-flow view because it is the only route by which tenant code reaches durable storage. There is no data path at all from tenant traffic to model training, and evaluation corpora are built from licensed material only.

How it is realised on Azure

The post-processing tier emits two Event Hubs streams; the code-bearing stream has no configured sink for a zero-retention tenant, so the emit is a no-op rather than a filtered write. A scrubber enforces the same rule at the Data Explorer and Blob sinks, which is where crash dumps and trace exporters would otherwise land. Retention mode is a column on the tenant row in Azure SQL, read through the policy cache, and failing closed to zero retention when the cache is cold. Consent and configuration changes are written to an append-only audit store kept five years, so a reviewer can see when the mode changed and who changed it.

Options weighed

• ChosenAbsent path for code-bearing telemetry, enforced at source and sink: A control that cannot be misconfigured and does not drift; costs the ability to diagnose a tenant-specific quality problem from data
• RejectedRetention flag checked before writing: The common design, and the one a reviewer is right to discount: one new code path, one debug flag or one unaware exporter defeats it
• RejectedRetain with encryption and customer-held keys: Strong on paper and still retention: the customer's code is in the platform's storage, which is the fact the review is actually about
• RejectedRetain anonymised or tokenised code: Source code is not anonymisable in any meaningful sense — identifiers, structure and comments carry the identity, and a fragment is often enough to identify the repository

Consequences

What it buys

• The privacy claim becomes structural rather than procedural, which is the form that survives an audit and a refactor
• Quality measurement still works, because retention and acceptance need character counts and offsets rather than characters
• A cold policy cache fails to zero retention, so an availability problem can never become a disclosure

What it costs

• A tenant-specific quality complaint cannot be investigated from retained examples, only from aggregates and a reproduction
• Evaluation corpora stay narrower than production traffic, so the offline gate is blind to constructs that only appear in customer code (ADR-08)
• Two telemetry streams and two enforcement points is more machinery than one flag, and the sink-side scrub must be re-verified whenever infrastructure changes

Choose differently when

A tenant that actively wants retention — to debug its own usage, or to build an internal corpus under its own legal basis — can opt in, and that is the diagnostic path. For a single-tenant sovereign deployment the whole question changes shape, because the retaining storage is the customer's own. The decision is about a shared multi-tenant platform holding other people's source code.

Why it holds up over time

A control implemented as a missing code path cannot be misconfigured, cannot drift, and does not need re-attesting after every refactor. As regulation around training data and code provenance tightens, the designs still adoptable will be the ones where the sensitive path was never built. That argument gets stronger with time, not weaker.

A filter on a 300 ms critical path will eventually be slow or unavailable. What happens to the suggestion then, and who decided?

Context

Two filters sit between generation and display. A secret scan catches keys, tokens and private-key blocks that the model may reproduce from its context or its training — the failure where a developer is handed back a credential, sometimes one of their own. A public-code match filter suppresses suggestions that reproduce a substantial verbatim span of known public source, which some enterprises treat as a licence exposure and others ignore. Both run inside a budget measured in single-digit milliseconds. In most systems what happens when such a filter is slow or down is never decided: it times out, and whichever way the timeout happens to be coded becomes the policy. For one of these filters failing open is a security incident and for the other it is a commercial preference, so a single timeout behaviour cannot be right for both.

Decision

Each filter declares its failure direction explicitly as part of its configuration, and the choice is a product decision rather than an implementation detail. The secret scan is fail-closed: if it cannot run, no suggestion is displayed. The public-code match filter is fail-closed for tenants that selected block mode and fail-open for tenants in warn or off mode. Every suppression is counted and attributed, so suppression is visible rather than silent, and a filter that starts suppressing a large fraction of candidates is an alert rather than a mystery.

How it is realised on Azure

Filters run in the stateless post-processing tier on Container Apps, in-process rather than as a network hop, so the budget is reachable. Failure direction, mode and thresholds are per-tenant configuration in Azure SQL read through the policy cache, defaulting to the stricter setting when the cache is cold. Filter time is reported separately from inference time in Azure Monitor. The public-code match index is refreshed in batch from a licensed corpus in Blob Storage and is never on the critical path for a tenant in off mode. Suppression counts by filter, tenant and reason go to Data Explorer alongside the interaction telemetry.

Options weighed

• ChosenPer-filter declared failure direction, counted suppressions: Makes the safety-versus-availability trade explicit per filter and per tenant; costs configuration surface and two code paths per filter
• RejectedOne uniform timeout policy for all filters: Simple and wrong for at least one filter by construction: the right answer differs between a credential and a licence preference
• RejectedFilter asynchronously, retract after display: A suggestion already displayed is a suggestion already read; retraction after the fact does not undo a disclosed secret
• DeferredPrevent at decoding time with constrained generation: Removes the post-hoc filter and its latency, and costs tokens-per-second on the critical path and a much harder correctness argument; worth revisiting as constrained decoding matures

Consequences

What it buys

• The one failure that cannot be allowed — handing back a credential — is structurally prevented rather than left to a timeout's default
• Tenants with different licence postures get different behaviour without different code paths in the filter itself
• Counted suppressions turn a silent quality cost into a measurable one, which is what makes the false-positive budget enforceable

What it costs

• A secret-scan outage is a full product outage for every tenant, which makes that component as availability-critical as inference
• Filtering spends part of a very small budget, and the spend is on the path of every displayed suggestion
• Each filter needs two tested code paths and a per-tenant configuration, which is a permanent increase in surface

Choose differently when

If generation-time prevention matured enough to make verbatim reproduction of secrets and licensed spans structurally impossible, the post-hoc filters could shrink to an audit sample and the fail-closed coupling would disappear. For a tenant with no licence concern and no secrets in its repositories — rare, and usually wrong about both — the filters could run in sample mode.

Why it holds up over time

"Every control on a critical path declares which way it fails, and the declaration is a product decision" is a rule that outlives any particular filter. It is also one of the few security properties that gets cheaper to keep as a system grows, because it is enforced by configuration shape rather than by vigilance.

A retrieval index is built once and read many times. Whose permissions does it serve — the builder's, or the reader's at the moment they read?

Context

Retrieval over the wider repository is what lifts a completion from the current file to the codebase, and it is the feature most likely to cause a disclosure. An index built under one set of permissions outlives those permissions: a contractor leaves a project, a repository is made private, a monorepo path moves behind an access list. If permission is captured at build time, the index keeps serving the old answer until it is rebuilt, and rebuilds take up to an hour for the largest workspace. The alternative is to resolve permission per request against the system that actually owns it, which puts a synchronous call to a third party inside a 300 ms budget — on a path where the obvious optimisation, caching the answer, is also the exact mistake being avoided.

Decision

Repository permission is evaluated at request time against the requesting principal, using the source hosting system as the authority. The default retrieval scope is the workspace, not the tenant. When the permission authority cannot answer inside the assembly budget, retrieval degrades to open-tabs-only context and the degradation is counted as a quality signal rather than treated as a silent fallback. Index contents are never served on the basis of permissions recorded at build time.

How it is realised on Azure

The retrieval service in the request plane calls source hosting with the principal from the validated Entra token and receives a scoped repository list, then retrieves only within that scope from the per-workspace indexes. Short-lived negative results may be cached; positive grants are not cached beyond the request. Private endpoints carry the call, and its latency is budgeted inside the server-side 15 ms assembly allowance with the degradation path taking over beyond it. Degradation-to-open-tabs is exported per tenant to Azure Monitor and correlated with acceptance in Data Explorer, so the cost of the degradation is visible in quality rather than only in latency.

Options weighed

• ChosenPer-request resolution against the owning authority, degrade on timeout: Revocation takes effect immediately and the index cannot outlive a permission; costs a third-party dependency inside the critical path
• RejectedPermissions captured at index build: Fast, cheap and wrong by up to a rebuild cycle — which is exactly the window in which a revocation matters most
• RejectedPer-principal index partitioning: Removes the request-time call and multiplies index count by the number of developers, which at 1.2 million makes the storage and rebuild cost absurd
• DeferredCached grants with a short TTL and push invalidation: The pragmatic middle, and it depends on an invalidation signal from a system the platform does not own; revisit if source hosting offers a reliable one

Consequences

What it buys

• A revocation takes effect on the next keystroke rather than on the next rebuild, which is the only version of this control that is defensible in a review
• Indexes become permission-free artefacts, so they can be rebuilt, moved and shared across a workspace's developers without re-deriving an access list
• The degradation path is counted, so the cost of a slow authority appears as a quality number rather than as an unexplained acceptance dip

What it costs

• A third-party call sits inside the assembly budget, and its tail latency becomes this platform's tail latency
• The degradation to open-tabs context is invisible to the developer and visibly worse in acceptance, so a slow authority quietly degrades the product
• Negative-result caching is the only caching allowed, which forgoes most of the available saving

Choose differently when

If source hosting offered a reliable permission-change event, cached grants with push invalidation would be strictly better: the same correctness with none of the critical-path cost. For a single-tenant deployment where every developer can read every repository, the check is vacuous and should be skipped rather than optimised.

Why it holds up over time

"Authorise against the authority at the moment of access" is as old as access control and keeps being re-learned by every system that builds a derived index over protected data. Search platforms, data catalogues, vector stores and retrieval layers all face it, and the ones that cached the grant all eventually disclosed something. That will not change.

More than half of what this fleet generates is never read. Which denominator makes that visible rather than invisible?

Context

The natural unit for an inference service is cost per request, or cost per thousand tokens. Both are actively misleading here. At a 55% cancellation rate, cost per request averages the cheap discarded work with the expensive useful work and produces a number that improves when the fleet generates more rubbish faster. Cost per token is worse, because it rewards verbosity. Neither denominator lets anyone answer the question an engineering leader and a tenant administrator both actually ask: what did the spend buy. And because cancellation is driven by client behaviour the platform does not control (ADR-05), a denominator that hides cancellation also hides the single largest lever on fleet cost.

Decision

The headline cost metric is cost per accepted suggestion, with cost per displayed suggestion reported alongside it. Wasted generation — accelerator seconds spent on suggestions that were cancelled or never displayed — is a named, capped, budgeted metric with an owner and an alert, targeted at no more than 18% of accelerator seconds. Cost is attributed to tenant, language and model so that model routing (ADR-14) can be argued on cost per accepted rather than on cost per token. Per-tenant spend ceilings degrade along the published ladder (ADR-06) rather than cutting off.

How it is realised on Azure

Accelerator seconds are attributed at the batch scheduler per sequence and tagged with tenant, model digest, language and request shape, then joined in Azure Data Explorer against the interaction outcomes that say whether the suggestion was displayed and accepted. The same figures feed the admin API and the nightly export to the billing platform, so the number an administrator reads is the number they are charged on. Eager cancellation reclamation (50 ms at p99) is the primary control on wasted generation; admission refusal (ADR-04) is the second, since a refused request costs nothing at all.

Options weighed

• ChosenCost per accepted suggestion, wasted generation capped: Denominated in what the customer buys and exposes the dominant waste term; requires joining cost telemetry to outcome telemetry
• RejectedCost per request: Hides the 55% that is discarded, and improves when the fleet wastes more work faster — an objective pointing the wrong way
• RejectedCost per thousand tokens: The vendor's unit, not the customer's; rewards verbosity and is not comparable across model sizes or accelerator generations
• DeferredCost per retained suggestion (survival-weighted): The truest denominator, matching the quality metric in ADR-07, and it arrives a day late; worth adopting for monthly reporting once the retention pipeline is trusted

Consequences

What it buys

• The dominant waste term becomes a named metric with a cap and an owner instead of an emergent property of the cancellation rate
• Model choice can be argued on cost per accepted suggestion, which is what makes "use the smaller model" a measurable claim rather than a preference
• Administrators are shown and billed on the same figures, which removes the most common source of distrust in a usage-based product

What it costs

• The metric requires joining cost telemetry to outcome telemetry, so a telemetry gap becomes a billing and reporting gap
• Acceptance rate sits in the denominator, so a quality regression shows up as a cost regression and the two have to be disentangled before either is acted on
• A tenant whose developers are simply fast typists looks expensive per accepted suggestion through no fault of the platform

Choose differently when

If cancellation fell to near zero — a surface where suggestions are explicitly requested rather than fired on a keystroke — cost per request becomes honest again and the extra machinery is unnecessary. The decision is forced by the cancellation rate, so it should be revisited if the dominant surface changes.

Why it holds up over time

Cost per accepted suggestion will outlive the accelerator generation, the model size and the pricing model, because it is denominated in the thing the customer is buying. Cost per request, per token and per GPU-hour all stop being comparable the moment any of those three change, which they do every year.

Single-line completions are the overwhelming majority and the cheapest; multi-line are rarer, more valuable and much more expensive. How many models serve that distribution?

Context

One mid-size model is operationally simple and uniformly priced, and it overpays for the common case while underperforming on the rare one. A deep cascade — try the smallest, escalate on a confidence signal — is cheapest at the median and pays twice at the tail, because an escalated request has already spent the small model's latency before the large one starts, inside a budget that has no room for it. There is also a measurement problem that is specific to this product: the counterfactual suggestion is never generated, so a wrong routing decision produces no evidence. If the fast model was routed a request the large model would have completed better, nothing in the telemetry says so — the developer simply did not accept, which looks identical to the model having nothing useful to offer.

Decision

Two models, routed once at admission on features available before generation: request shape (single-line versus multi-line intent), language, file size and tenant tier. No mid-generation escalation, because escalation spends the budget twice. Routing is configuration rather than code, and routing quality is evaluated by holding a sample of traffic on forced routes so that the counterfactual exists for the sample even though it never exists for an individual request.

How it is realised on Azure

A fast single-line server and an escalated multi-line server run as separate AKS GPU node pools, each with its own affinity map and prefix cache, so a session's warm state belongs to one pool. The routing rule is versioned configuration in Azure SQL read through the policy cache, evaluated by the affinity router before admission so that placement and model choice are one decision. A small percentage of traffic is assigned to forced-route arms through the same experiment machinery as model promotion (ADR-09), which is what makes routing quality measurable at all.

Options weighed

• ChosenTwo models, single routing decision at admission, forced-route sampling: Keeps the common case cheap without spending the budget twice, and creates the counterfactual for a sample; costs two fleets to operate and two working sets to hold
• RejectedOne mid-size model for everything: Simplest to operate and run; overpays on every single-line completion, which is most of 2.4 billion daily requests
• RejectedSmall-first cascade with mid-generation escalation: Cheapest at the median and unaffordable at the tail: the escalated request has already spent part of a 300 ms budget before the larger model begins
• DeferredClient-side routing from editor context: Saves a server-side decision and puts a cost-bearing choice in an untrusted client that ships on its own cadence; revisit only with the server retaining a veto

Consequences

What it buys

• The majority case is served by the cheap model, which is the single largest structural saving available after prefix reuse
• Routing and placement are one decision, so a request is never routed to a model whose pool does not hold its warm prefix
• Forced-route sampling creates the only evidence that can show the routing rule is wrong

What it costs

• Two GPU pools means two working sets, two rollout surfaces and two sets of capacity decisions (ADR-15)
• A session that changes shape mid-edit may move pools and pay a cold prefix, so the routing feature set has to be stable within a session
• Routing quality is only measurable on the forced-route sample, so the rule is always evaluated on a small fraction of the evidence

Choose differently when

If one model could serve both shapes at the fast model's cost, two pools stop earning their complexity and should be collapsed immediately — this is a decision that should be re-tested at every model generation. Conversely, if a genuinely cheap confidence signal emerged before generation, a three-tier portfolio could beat two; the barrier is that the signal must be available at admission, not mid-stream.

Why it holds up over time

The durable part is the refusal to make a routing decision that cannot be evaluated: building the forced-route sample is what keeps the rule honest as the model portfolio changes. Routing by request shape rather than by mid-flight escalation is also a property of the latency budget rather than of the models, so it holds as long as the budget does.

Cross-continental round-trip time alone can consume most of a 300 ms first-token budget. How many fleets does that imply, and what is the cost of the answer?

Context

A developer in Singapore served from Western Europe spends 150 to 200 milliseconds of the budget on network before a token is generated, which leaves nothing for assembly, admission and decoding. The latency contract therefore requires inference near the developer. But the accelerator fleet is the expensive, lumpy resource: splitting four thousand accelerators across three regions means three smaller fleets that batch less efficiently, three working sets rather than one, and three rollout surfaces for every model change. It also interacts badly with session affinity (ADR-01), because the warm-prefix working set is exactly the thing that fragments. Meanwhile geography pinning for data residency (ADR-10) forces some of the split regardless of latency, which means the question is not whether to split but whether to pretend the split is only about latency.

Decision

Each serving region is a complete working set: its own GPU pools, its own prefix caches, its own policy read replica and its own measurement path. Prefix caches are never replicated or transferred across regions — they are recomputable, so replicating them would couple regions for no durability gain. A region loss costs cold starts rather than correctness: sessions re-establish elsewhere with cold caches, inside a 5-minute RTO. The control plane has one write region with regional read replicas; the model registry is geo-replicated because weights are large, immutable and read everywhere.

How it is realised on Azure

Azure Front Door routes by geography and by tenant residency pinning rather than purely by latency, refusing rather than spilling when a tenant's declared geography is unavailable. Each region runs its own AKS GPU node pools, Container Apps request plane, and Redis affinity and policy caches; Azure Container Registry is geo-replicated so weights are pulled locally by digest. Azure SQL has one write region with read replicas per serving region, and Event Hubs plus Data Explorer are regional with aggregates consolidated asynchronously. Measurement is pinned to a separate zone within each region so a telemetry incident and a serving incident are not the same incident.

Options weighed

• ChosenThree complete regional working sets, no cache replication: Holds the latency contract and keeps a region loss to cold starts; pays with fleet fragmentation, lower batch efficiency and three rollout surfaces
• RejectedOne global fleet behind a global edge: Best batch efficiency and one working set, and it cannot meet a 300 ms p95 for anyone more than a continent away
• RejectedRegional request planes, centralised inference: Looks like a compromise and keeps the whole network cost on the critical path, which is the part of the budget that cannot be optimised
• RejectedCross-region prefix cache transfer on failover: Couples regions and adds a transfer cost for state that can simply be recomputed — the wrong trade for a cache that is not a system of record

Consequences

What it buys

• The latency contract is attainable for developers on three continents rather than one, which is the difference between a product and a demo
• A region loss is a cold-start event with a 5-minute RTO, because nothing in the region held authority over anything
• Data residency and latency are served by the same topology, so pinning is not an exception to the architecture

What it costs

• Three smaller fleets batch less efficiently than one large one, which is a permanent utilisation cost
• The warm-prefix working set fragments, so each region needs its own headroom and the sum exceeds what one fleet would need
• Every model promotion, rollback and configuration change has three surfaces, which multiplies the rollout machinery and the ways a ramp can be inconsistent

Choose differently when

If network latency to a central region fell below roughly 30 milliseconds for the served population — a single-country deployment, or a future where that is simply true — one fleet becomes better on every axis and the regions should be collapsed. The sovereign single-tenant deployment in phase 3 is the opposite case: a region of one tenant, where fragmentation is the whole point and efficiency is explicitly not the objective.

Why it holds up over time

"Replicate authority, recompute derived state" is the transferable rule, and it is what keeps a multi-region design simple: the only things that cross a region boundary are the small immutable ones. That will hold through any change in accelerator, model or cloud. What will change is the number of regions, which is a function of network physics and of where the customers are.