Conformant to everything except the decision

A conformance report is a small file with a large social function. The one merged into the Gateway API Inference Extension repository on 3 September records that NGINX Gateway Fabric v2.7.0 was run against version v1.6.0 of the extension. Twelve tests passed, two were skipped, none failed. The report stamps itself 2026-09-02T19:25:03Z and, because of the two skips, grades the core profile result: partial. Two more reports landed in the same fortnight, from Alibaba's ACK Gateway and from Higress. After a year in which every vendor built its own way of pointing a proxy at a fleet of GPUs, the ecosystem has begun certifying itself against a common standard.

It is worth reading what the certificate actually attests. The suite's central data-path test is described in its own source as verifying that the "Inference gateway should send traffic to an endpoint in the list returned by EPP". EPP is the Endpoint Picker: the component that looks at every replica of a model server and decides which one should handle this request. The test does not ask whether that decision was any good. It asks whether the gateway did as it was told.

That is the whole shape of the thing, and it is not an oversight. The standard specifies where to ask for a routing decision. It does not specify how to make one. Conformance therefore certifies obedience rather than competence — and in June, the component that supplies the competence moved out of the repository altogether.

The details are worth having precisely, because they are unusually legible. All of this is read from the projects' own repositories: the commits, the API types, the test descriptions and the reports themselves.

The API that remains under kubernetes-sigs is InferencePool, and it is a modest object. Its spec carries a label selector saying which pods are members of the pool, a list of target ports, and a reference to an Endpoint Picker service — a name, a port, and a failureMode. That is close to the entire surface. Read it as prose and the standard says: here is a set of GPUs, here is the network address of the thing that will choose between them, and here is what to do if that thing stops answering.

The choosing itself now lives in a repository called llm-d-router, under the llm-d organisation. Its README is candid about the division: core Endpoint Picker code and the InferenceObjective and InferenceModelRewrite APIs "have been merged into this repository", and the Kubernetes repository "now exclusively hosts the InferencePool API". It describes the Endpoint Picker as "the intelligent routing engine that serves as the 'brain' of the router", weighing KV-cache locality, current load and priority. The corresponding subtraction is a single commit on 17 June: Remove inference objective, model rewrite and endpoint picker config APIs. The Kubernetes-side README now notes that the moved packages will accept no new code and "will be archived soon", and that what stays is the InferencePool API, a deliberately minimal reference picker, and the conformance tests.

The contrast in the last fortnight is stark. In the standards repository: three conformance reports, dependency bumps, and fixes to the lightweight picker that exists so the conformance tests have something to run against. In llm-d-router, on 4 and 5 September alone: folding prefix matching into a single accumulator, ordering request-control hooks by a data-dependency graph, enforcing boolean values for priority routing, adding affinity filter composition requirements. One repository is issuing certificates. The other is changing, daily, the thing the certificates decline to measure.

To see why this is more consequential than the usual separation of interface from implementation, you have to appreciate how strange LLM routing is. Ordinary load balancing survives on ignorance. Round-robin works because requests are roughly interchangeable and backends are roughly identical, so a balancer needs to know almost nothing about what a server is holding. Inference breaks both halves of that. Requests are wildly unequal in cost, and replicas are not interchangeable, because each one holds a different KV cache — the retained attention state from earlier tokens. Route a request to a replica that already has its prefix cached and the prefill work is skipped. Route it elsewhere and the same work is done again on a different GPU. The balancer's ignorance, once a virtue, becomes the largest line item in the bill.

So the picker has to see inside the server. The project's own model server protocol document sets out what a model server must expose for this to work: queued requests, running requests, KV cache utilisation, and — for adapter-aware routing — a gauge listing which LoRA adapters are resident in GPU memory and which are waiting. It gives the vLLM metric names alongside the equivalents for Triton, TensorRT-LLM and SGLang. It also concedes something that standards rarely admit in writing: the reference picker's LoRA algorithm "is highly biased towards vLLM's current dynamic LoRA implementation."

That sentence is the honest core of the whole arrangement. A routing decision good enough to be worth making is coupled to the internals of a particular model server, and those internals change faster than any standards body can ratify. Faced with that, the Kubernetes project standardised what could hold still — the pool, the reference, the failure mode — and let the rest go somewhere it could move.

It also relocated the decision in a second sense, which matters more than the first. InferencePool sits in a vendor-neutral home with the governance and review that implies. The picker now sits in a project whose alignment is explicit rather than hidden: the Kubernetes README announces that Inference Gateway "has partnered with vLLM to accelerate LLM serving optimizations with llm-d". Nobody is being deceived here — the partnership is advertised in the first screen of the README, and putting the picker where the model server developers actually are is a defensible reason for the move. But the effect is that the neutral body kept the part with no performance consequences, and the part that decides performance went to live nearer one model server. The admission about LoRA bias and the choice of new home are the same fact seen from two angles.

The strongest case for all this is that it is simply how good standards behave. SQL standardises a language and leaves the query planner to competition; nobody wants a ratified join-ordering algorithm, and the planner is where databases earn their keep. Gateway API itself standardises resources rather than proxy algorithms. Draw the line narrowly, and implementations stay free to be excellent. Draw it widely and you freeze a bad idea into a version number for a decade. On that reading, the June split was not a retreat. It was a standards body declining to specify something it had no business specifying, and the three conformance reports are exactly the payoff: a pool definition you can write once and point at NGINX, or Istio, or a cloud load balancer.

The analogy has one load-bearing flaw. A query planner ships inside the database you already chose; its excellence is part of what you bought. Here the standard makes the decision-maker a separately built, separately governed process sitting in the request path — and then defaults the pool's failureMode to FailClose, meaning that when the picker stops answering, requests are dropped rather than routed anyway. The suite does test the opposite behaviour, fail-open, but that is the setting you must ask for. The portable object therefore carries a hard runtime dependency on a component the standard deliberately does not describe, and the safe-looking default makes that component's availability your availability.

This is what makes "conformant" a slippery word here. Two conformant gateways, fed the same InferencePool, will give you different tail latency, different tokens per GPU-hour and different effective capacity, because they are asking different brains. Your manifests are portable. Your performance is not. Of the fourteen tests in the suite, most check that the API object is accepted or rejected correctly — invalid references, missing picker references, port validation, status conditions. That is genuinely useful work; those are the failures that waste an afternoon. It is not the work that decides whether an inference platform is affordable.

The generalisable point is that what a standard declines to specify is a fairly reliable map of where an industry expects to compete. Read that way, the inference gateway standard is an unusually frank document. It says: the plumbing is settled, and the decision is contested. Everything about cost, latency and capacity lives on the contested side.

Architects should adjust accordingly, and mostly in procurement. A conformance report is about to start appearing in vendor material, where "certified conformant" will be read as "equivalent". It does not mean that. The useful question in an evaluation is no longer whether a gateway passes, but which picker it ships with, who maintains it, how it behaves when the model server it was tuned against changes its metrics, and what happens to your traffic when it stops responding. None of those appear in the YAML.

There is a particular irony in the fact that the reference picker still in the Kubernetes repository exists chiefly so the conformance tests have something to talk to. The standard kept a brain only smart enough to prove the wiring works. That is a reasonable engineering decision and an unusually clear statement of where the value went. The industry has spent a year building the part everyone can agree on.