The queue is part of the answer

Two requests. Same model, same weights, same prompt, greedy decoding. One arrives at three in the morning and has the server to itself. The other arrives at two in the afternoon and is admitted into a batch alongside thirty other people's work. There is no guarantee the two come back with the same text, and on a well-loaded fleet they routinely do not.

Anyone who has written a reduction kernel has known this for years. What happened this month is that somebody wrote the number down.

On 4 September a vLLM contributor filed an RFC with an unusually candid title: "[RFC] Length-aware batch composition for admission scheduling — experimental evidence, fairness fix, and where it breaks". The proposal itself is a scheduling idea — pair long and short requests inside a scheduling window so that they finish together, an LPT-style interleaving meant to improve fairness and utilisation when capacity is tight. The experimental section is the reason to read it. Under capacity pressure the policy delivers a makespan reduction of 7.3% and a throughput gain of 7.8%. Where the request pool is relaxed it does nothing measurable. On workloads that end at EOS rather than at a uniform max_tokens, it is actively harmful: makespan up by 38 to 49 per cent. The length estimators the whole scheme depends on reach a Spearman correlation of roughly 0.27 to 0.34, which is to say they barely work.

A proposal filed with the evidence against it, in public, by the person proposing it. That is worth more than most announcements.

But the number that matters is not in the performance table. Changing which requests shared a batch caused 96.6% of completions to differ byte-wise from the first-come-first-served baseline. The RFC explains it in one clause — "floating-point non-associativity means the same request can produce different logits" — then concedes that the policy "cannot hold a batch-invariance guarantee" and scopes itself to "throughput-oriented batch workloads where cross-run output identity is not required at all."

Read that as a property of the proposal and it files away as a niche caveat. Read it as a property of the system and the architecture looks different. FCFS was never the deterministic baseline being departed from. FCFS is just another batch-composition policy, one whose composition is a function of arrival order instead of predicted length. This RFC did not introduce nondeterminism into vLLM. It swapped one scheduling policy for another, measured how much of the output moved, and found that nearly all of it did.

Which is the point. In an inference server the scheduler is not transport. It belongs to the output contract.

We do not usually draw it that way. The reference diagram every team sketches has a box called "the model", which is where behaviour lives, and a set of boxes around it — router, queue, scheduler, KV cache, kernel backend — which are where performance lives. The architectural convention is that you may tune the outer boxes freely because they move latency and cost around without touching meaning. That convention is load-bearing. It is why batch size, chunked prefill thresholds, tensor-parallel degree and admission policy sit in a platform team's Helm values rather than in a change-controlled interface, and why nobody from the model side reviews them.

The convention is also wrong, and the industry has already admitted as much in code. vLLM ships a mode whose entire purpose is to restore the guarantee the convention assumes for free. Set VLLM_BATCH_INVARIANT=1 and, in the project's own words, "the output of a model is deterministic and independent of the batch size or the order of requests in a batch." The machinery under it descends from Thinking Machines Lab's batch_invariant_ops, which substitutes fixed-reduction-order implementations of torch.mm, torch.addmm, torch.log_softmax and torch.mean so that each batch element computes the same way regardless of how many neighbours it has. The documentation is blunt about the trade: enabling it "may impact performance compared to the default non-deterministic mode. This trade-off is intentional to guarantee reproducibility."

It does not say by how much, and that omission is the interesting part. An independent benchmark published on GitHub during this same fortnight went and measured it: Qwen3-1.7B on a single RTX 4090, vLLM 0.28.0, CUDA graphs enabled, 128 tokens per request, medians of five timed runs. Invariant mode cost between 53.6% and 66.6% of throughput depending on batch size — 11,152 tokens per second falling to 5,176 at batch 64, 226 falling to 75 at batch 1. One small model on one consumer card is not a fleet figure and should not be quoted as one. What makes it instructive is its own commit log: the headline began at "27 to 30 percent", then was corrected twice on the same day to 54–67% once CUDA graphs were enabled to make the measurement deployment-realistic. Even the cost of determinism is configuration-dependent.

And the guarantee, once bought, is conditional. On 11 September another contributor filed a bug: batch invariance breaks when sequence parallelism or async tensor parallelism is switched on. With TP=4 and SP enabled, across 64 greedy prompts, 1,522 of 1,536 tokens showed differing logprobs, 17 of 64 prompts produced different greedy output, and the maximum logprob delta was 2.98. At TP=2, only 4 tokens of 1,536 differed. The failure tracks which rank owns which output chunk — so whether your determinism flag actually holds depends on your parallelism topology, and it fails silently. There is no error. There is just different text.

Put the three together and the shape is clear. The default is nondeterministic. The fix is opt-in, undocumented in cost, and priced in the currency an infrastructure team is measured on. And the fix has a hole in it that only appears in a particular parallel configuration that a capacity decision might introduce next quarter.

The counterargument is strong and deserves stating at full strength. Language models are sampled. Most production traffic runs at temperature above zero, where run-to-run variation is not merely tolerated but deliberate. Nobody signed a contract promising bit-identical completions, users cannot tell two acceptable paraphrases apart, and spending half a fleet to buy a property the product does not need is a bad trade made by someone who has confused reproducibility with correctness. On the evidence above, the honest conclusion for most teams is to leave the flag off.

I agree with nearly all of that, and it does not touch the problem. The argument is not that everyone should turn determinism on. It is that the property is currently undeclared, unowned, and coupled to load.

Coupled to load is the phrase to sit with. The divergence is not random noise sprinkled evenly across time; it is a function of what else was in the machine. It is largest exactly when the fleet is busy, which is exactly when incidents happen. That makes a whole class of ordinary engineering work quietly unsound. An evaluation suite run against a quiet staging endpoint measures a numerical regime that production never occupies. An A/B test between two prompts on a shared fleet has each arm's neighbours as an uncontrolled variable. A regression triage that begins "I can't reproduce it" may be literally correct and still wrong about the cause, because reproducing it would require reproducing the queue. And an RL loop that scores its own generations inherits, as reward signal, whatever the scheduler was doing that afternoon.

None of these is a hypothetical failure of the serving layer. They are the normal uses of a serving layer, running on an assumption the serving layer no longer meets.

One limit on all of the above is worth naming. Every piece of evidence here comes out of a single project's public record: vLLM's issue tracker, vLLM's documentation, a benchmark run against vLLM, and the kernel library vLLM's invariant path descends from. I have not verified how SGLang, TensorRT-LLM or any hosted endpoint behaves, and none of them owes it to anyone to behave the same way. What makes the narrow evidence worth generalising from is that vLLM is the open reference the rest of the field reads: its scheduler is the one most widely copied, and its willingness to publish its own negative results is the reason these numbers exist at all. The absence of equivalent numbers elsewhere is not evidence of a better guarantee. It is an absence of measurement.

This is not an isolated fact about batching, either. A separate vLLM RFC filed on 29 August notes that blocks written to the filesystem KV offload tier "are never checksummed", so a corrupted block "is copied into the primary tier and consumed as attention KV, producing wrong logits with no error signal." Different mechanism, same category: a piece of infrastructure chosen for throughput, sitting inside the path that determines what the model says, with no signal crossing the boundary when it changes the answer.

The discipline has a well-worn move for shared resources that interfere with each other. We isolate them, or we declare the interference and price it. CPU shares, IOPS limits, connection pools, noisy-neighbour SLOs, cgroups — the whole apparatus of multi-tenancy exists because one tenant's load changes another tenant's experience. The batch is the newest shared resource in the stack and it has none of that apparatus: no isolation primitive, no declared class of service, no line in any API description, no place in any SLO document where an architect could write down which guarantee this endpoint sells.

The reason it has escaped is that every other shared resource we learned to govern changes only when you get your answer. This one changes what the answer is.

That is a new kind of coupling, and it has been sitting in production for two years with a scheduler tuning flag as its only control surface.