Your Answer Depends on Who Else Is in the Batch

Send the same prompt to Qwen3-235B-A22B one thousand times at temperature 0. Greedy decoding, no sampling, no seed involved. You get eighty distinct completions, and the first divergence appears at token 103: 992 samples continue with "Queens, New York" and 8 with "New York City" (He and Thinking Machines Lab, 2025, Defeating Nondeterminism in LLM Inference).

Temperature 0 is deterministic by construction. Take the argmax, emit it, repeat. So the nondeterminism is not in the sampler; the logits themselves are different between runs. Every engineer who has hit this reaches for the same explanation, which is that GPUs are concurrent and floating-point addition is not associative, so atomicAdd accumulates in an unpredictable order. It is a satisfying story about a mechanism that is barely present. A typical LLM forward pass contains no atomic adds at all, and running the same kernel on the same batch twice is bit-identical.

The actual cause is stranger and more consequential. Kernels choose how to split their reductions based on the shape of their inputs, and the shape includes the batch. Your request is processed alongside whatever unrelated traffic arrived in the same scheduling window, so the summation order inside every matmul, every RMSNorm, and every attention call is a function of other people's requests. Nothing about your input changed. The order of additions did, and floating-point arithmetic notices.

Why this matters: Reproducibility is the obvious casualty, and it is the smaller one. If the sampler and the trainer in a reinforcement-learning loop compute different logprobs for the same tokens, an algorithm you designed to be on-policy is quietly off-policy, and the correction that hides the discrepancy can fail outright. Determinism here is a correctness property of the training algorithm, not a debugging convenience.

TL;DR

• Greedy decoding is deterministic; the forward pass that feeds it is not. Qwen3-235B produced 80 distinct completions from 1,000 temperature-0 samples of one prompt, first diverging at token 103.
• The usual explanation, nondeterministic atomicAdd, is wrong for this workload. Run the same kernel on the same batch twice and it is bit-identical.
• The real dependency is batch invariance: kernels pick a parallelisation, and therefore a reduction order, from input shape. The same matmul reduces differently at batch 1 and batch 32.
• Three reduction-heavy operations carry almost all of it: matmul, RMSNorm, and attention.
• For attention the fix is a fixed split size, not a fixed split count. A fixed count makes chunk boundaries move as the KV cache grows during decoding, so a request's reduction tree changes shape mid-generation.
• The cost has now been measured twice independently. Thinking Machines: 26 s to 55 s unoptimised, 42 s with a better kernel. SGLang across three attention backends: 34.35% average slowdown, spanning 24.4% to 55.1%.
• torch.use_deterministic_algorithms(True) does not fix this. It buys run-to-run determinism at fixed shape. Batch invariance is a different property, and tensor-parallel invariance is a third.
• With batch-invariant kernels, the KL divergence between an RL sampler and its trainer sits flat at zero, against roughly 0.001 under off-policy correction. Without any correction, one run collapsed in reward around step 318.

At a Glance

flowchart LR
  Q["Your request"] --> SCH["Continuous batching<br/>scheduler"]
  OTH["Unrelated traffic"] --> SCH
  SCH --> SHAPE["Batch shape<br/>this iteration"]
  SHAPE --> KSEL["Kernel picks split<br/>strategy from shape"]
  KSEL --> ORD["Reduction order"]
  ORD --> BITS["Last bits of every logit"]
  BITS --> TIE["Two near-tied<br/>candidates swap"]
  TIE --> DIV["Completion diverges<br/>from here on"]
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  class Q,OTH blue
  class SCH,SHAPE slate
  class KSEL,ORD,BITS purple
  class TIE,DIV rose

The causal chain has exactly one link that anyone would call surprising, and it is the fourth one.

Before Anyone Checked

Nondeterminism in numerical computing is not new, and neither is the standard that makes it inevitable. What is new is that the output of the computation became a discrete token sequence, which turns a difference of \(10^{-7}\) into a difference of several paragraphs.

timeline
    title From rounding modes to reward collapse
    1985 : IEEE 754 standardises binary floating point
         : Addition is commutative but not associative; order changes the result
    2016 : cuDNN autotuning benchmarks algorithms per shape
         : Convolution results start depending on which algorithm won that day
    2020 : PyTorch adds a global deterministic-algorithms switch
         : Fixes run-to-run variation at fixed shape; nothing about shape changes
    2022 : Orca introduces iteration-level continuous batching
         : A request's batch neighbours are now decided by unrelated arrival times
    2023 : vLLM and PagedAttention make continuous batching the default everywhere
         : Effective batch size becomes a function of traffic
    2024 : RL from verifiable rewards makes sampler-trainer logprob agreement load-bearing
         : Practitioners start reporting unexplained train-inference mismatch
    2025 : Thinking Machines names batch invariance and ships batch-invariant kernels
         : SGLang reproduces it across FlashInfer, FlashAttention 3 and Triton
         : Follow-up work extends invariance across tensor-parallel sizes

The 1985 fact is the one everything else rests on. IEEE 754 defines each operation to round correctly, which makes a single addition perfectly reproducible and makes a sequence of additions dependent on its grouping. \((a + b) + c\) and \(a + (b + c)\) are both correct and they are not the same number.

For decades this was a numerical-analysis concern with a bounded blast radius. If your simulation's tenth decimal place moves, you round the report. The change came when the consumer of the arithmetic stopped being a physical quantity and started being an argmax, at which point a difference below the noise floor either changes nothing at all or changes everything downstream of it, with no middle ground.

[IMAGE: Two summation trees over the same eight values, one left-to-right sequential and one balanced binary. Each node is annotated with its fp32 intermediate value; the two roots differ in the final three bits, shown in binary. Caption: "Both trees are correct. IEEE 754 guarantees each individual rounding, not the result of a particular grouping."]

Why the Same Kernel Gives Two Answers

Non-associativity, made concrete

Take one value of 1.0 and one thousand values of \(3 \times 10^{-8}\), all in fp32.

The unit in the last place at 1.0 is \(2^{-23} \approx 1.19 \times 10^{-7}\), so round-to-nearest snaps \(1.0 + x\) back to exactly 1.0 whenever \(x < 5.96 \times 10^{-8}\). Since \(3 \times 10^{-8}\) is below that threshold, a sequential accumulator that starts at 1.0 and adds the small values one at a time returns exactly 1.0. Every single addition is individually correct and every single one is absorbed.

Now split the work: sum the thousand small values among themselves first, giving \(3 \times 10^{-5}\), then add 1.0. The result is 1.0000300. The two answers differ by \(3 \times 10^{-5}\), which is about 252 units in the last place.

Nothing pathological happened. This is ordinary summation of ordinary numbers, and the entire difference is which values met each other first. A GPU kernel decides that by choosing how many threads participate in the reduction, which it decides from the shape of the tensor.

[IMAGE: Two accumulator traces plotted against step number for the same 1,001 values. The sequential trace is a flat line pinned at exactly 1.0 for all 1,000 steps; the split trace climbs linearly to 3e-5 and then jumps to 1.00003. A callout marks the half-ULP threshold at 5.96e-8. Caption: "Absorption is not an error. Every addition on the flat line rounded correctly."]

Where the reductions are

Only operations that sum along an axis can exhibit this, which narrows the surface considerably. Three matter.

Matmul. Every output element is a dot product, a reduction over the contraction dimension. For a large contraction, one thread cannot do it alone, so the kernel splits the sum across threads or blocks and combines partial results. How many partials, and in what order they combine, is chosen per shape by the autotuner.

RMSNorm. A sum of squares across the hidden dimension per token. The kernel decides how many rows to assign to each block, and when the batch is small it may assign several rows per block to keep the GPU busy, changing the per-row reduction layout.

Attention. A sum over the KV length, which is neither fixed nor known at compile time and which grows by one on every decode step. Splitting it is mandatory for long contexts and is where the subtlest of the three failures lives.

flowchart TB
  subgraph SMALL["Small batch"]
    S1["Few output rows"] --> S2["Split reduction across many blocks<br/>to fill the GPU"]
    S2 --> S3["Partial sums combined in a tree"]
  end
  subgraph LARGE["Large batch"]
    L1["Many output rows"] --> L2["One block per row<br/>GPU already full"]
    L2 --> L3["Single sequential accumulator"]
  end
  S3 --> R["Different rounding<br/>same mathematics"]
  L3 --> R
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  class S1,S2,S3 teal
  class L1,L2,L3 amber
  class R rose

The behaviour in that diagram is not a bug and is not something a kernel author should be talked out of. At small batch there are not enough output rows to occupy every streaming multiprocessor, so splitting the reduction is the only way to get parallelism. At large batch the rows themselves provide it, and splitting would add a needless combination step. The autotuner is doing its job. Its job happens to be incompatible with reproducibility.

The attention detail that gets missed

Attention's reduction runs over the KV cache, whose length changes on every decode step. Two ways to split it look equivalent and are not.

A fixed split count divides the cache into, say, four chunks whatever its length. At length 1,000 the chunks are 250 elements; at length 1,024 they are 256. The key at index 300 therefore accumulates with a different set of neighbours at different points in the same generation, so a request's own reduction tree changes shape as it runs.

A fixed split size divides into constant-size chunks and lets the count vary. A 1,000-element cache becomes three chunks of 256 and one of 232; a 1,024-element cache becomes four of 256. The key at index 300 is always in the chunk beginning at 256, always accumulating with the same neighbours in the same order. The reduction strategy is now independent of how many query tokens happen to be in flight, which is the property the whole exercise is after.

That distinction is the substance of making attention batch-invariant, and it is the reason a naive attempt at deterministic attention often fails while looking correct.

[IMAGE: Two horizontal strips representing a KV cache, one of length 1,000 and one of length 1,024. Top pair uses a fixed split count of four, so boundaries fall at 250/500/750 and 256/512/768 and the element at index 300 lands in differently-composed chunks. Bottom pair uses a fixed split size of 256, so boundaries fall at 256/512/768 in both and index 300 sits in the same chunk with the same neighbours. Caption: "The same index, the same request, two different accumulation partners, purely because the cache grew."]

Why a server amplifies it

A modern inference server does not process requests one at a time. Continuous batching admits and retires sequences at iteration granularity, so the number of sequences in flight rises and falls with arrival rate. Your request might run in a batch of 3 at 04:00 and a batch of 47 at 14:00, and the model executes different reduction trees in each case.

This is what makes the failure so hard to reproduce locally. On a developer laptop with batch size 1, everything is deterministic and everything is fine. The nondeterminism is a property of the deployment, and it appears exactly where you cannot attach a debugger.

Seeing It in Motion

sequenceDiagram
    participant U as Client
    participant S as Scheduler
    participant K as Attention kernel
    participant D as Sampler
    U->>S: prompt, temperature 0
    Note over S: 6 other requests in flight
    S->>K: forward, batch shape (7, ...)
    K->>K: choose split strategy from shape
    K-->>D: logits, low bits set by this order
    D-->>U: token 103 is Queens
    U->>S: identical prompt, later
    Note over S: 31 other requests in flight
    S->>K: forward, batch shape (32, ...)
    K->>K: different split, different order
    K-->>D: logits differ by ~1e-6
    D-->>U: token 103 is New
    Note over D: argmax flipped, so every later token differs

Two things in that trace deserve attention. The kernel is not misbehaving at any point, and the sampler is not misbehaving at any point. The divergence enters through a scheduling decision that neither component can see, and it becomes visible only when two candidate logits are close enough that a \(10^{-6}\) perturbation reorders them. That happens rarely per token and reliably over a thousand tokens.

[IMAGE: Histogram of the gap between the top-1 and top-2 logits across a full generation, log-scaled x-axis, with a shaded region at the left marking gaps below 1e-5. An annotation counts how many tokens fall in the shaded region. Caption: "Most tokens are decided by a comfortable margin. The handful that are not decide everything after them."]

By the Numbers

Determinism is not free and both published implementations report the price.

And the determinism actually obtained:

Sources: vLLM timings, the 1,000-sample Qwen3-235B experiment, and all three RL figures are from Thinking Machines Lab's September 2025 post (He, 2025). SGLang overheads and unique-output counts are from the LMSYS engineering post of 22 September 2025 (LMSYS, 2025). The two sets of overhead numbers are not directly comparable: they use different models, different output lengths, and different attention backends. Read them as two independent estimates of the same order of magnitude, roughly a third to double, rather than as competing measurements of one quantity.

A Concrete Example

Follow a single logit through one RMSNorm to see the failure appear and then propagate.

Step 1 — the tensor. One token, hidden size 4,096, activations in fp32. RMSNorm needs \(\sum_{j=1}^{4096} x_j^2\). Suppose one coordinate is large, \(x_1^2 = 1.0\), and the remaining 4,095 are small, each contributing \(3 \times 10^{-8}\) to the sum of squares. The exact mathematical total is \(1.00012285\).

Step 2 — large batch, one block per row. With 512 tokens in the batch there are plenty of rows to fill the GPU, so the kernel gives each row to one block and accumulates sequentially from index 1. The accumulator holds 1.0 after the first term. Each subsequent addition of \(3 \times 10^{-8}\) falls below half the ULP at 1.0, which is \(5.96 \times 10^{-8}\), so each is absorbed and the accumulator never moves. Final value: 1.0000000.

Step 3 — small batch, split reduction. With 2 tokens in the batch there are not enough rows to fill the GPU, so the kernel splits each row's reduction across 32 blocks of 128 elements and combines the partials in a tree. Within block 0, a warp-shuffle tree pairs the 127 small values with each other before any of them meets the large one, so they survive as \(3.81 \times 10^{-6}\) and block 0 returns 1.0000038. The other 31 blocks each accumulate 128 small values to \(3.84 \times 10^{-6}\) with no large value present to absorb them, contributing \(1.190 \times 10^{-4}\) between them. Final value: 1.0001228.

Step 4 — the divergence, in relative terms. The two sums differ by \(1.228 \times 10^{-4}\), a relative error of about \(1.2 \times 10^{-4}\). RMSNorm divides by the square root of the mean, so the scale factor differs by roughly \(6 \times 10^{-5}\) relative, and every one of the 4,096 activations leaving this layer is off by that fraction.

Step 5 — through the stack. That perturbation propagates through the remaining layers. It neither amplifies nor cancels in any systematic way; it stays a relative perturbation on the order of \(10^{-5}\) to \(10^{-6}\) by the time it reaches the logits, which is consistent with the divergence behaviour reported in practice.

Step 6 — the argmax. At most decode steps the top two logits differ by a comfortable margin and a \(10^{-6}\) shift changes nothing. At a step where the model is genuinely undecided, say logits of 12.847312 and 12.847309 for " Queens" and " New", the shift is larger than the gap and the ranking flips.

Step 7 — no recovery. The emitted token becomes context for every subsequent step. There is no mechanism that pulls the two runs back together, which is why the Qwen3 experiment shows agreement through token 102 and complete divergence from token 103 onward. One flipped comparison out of roughly a hundred thousand produced eighty distinct thousand-token completions.

Steps 2 and 3 are the whole argument. The mathematics is identical, the hardware is identical, the weights are identical, and the only thing that changed was how many other requests were in the batch.

Where It Breaks

[IMAGE: A nested-rings diagram of four determinism properties, each ring strictly containing the ones inside it: "same call twice" (innermost, covered by seeds and framework flags), "same input, any batch size" (batch invariance), "same input, any parallelism degree" (TP invariance), "same input, any serving configuration" (outermost, unsolved). Each ring is annotated with the mechanism that achieves it and the mechanism's cost. Caption: "Each ring is a strictly stronger guarantee, and the tools stop before the outermost one."]

Framework determinism flags solve a different problem

torch.use_deterministic_algorithms(True) selects deterministic implementations where they exist and raises an error where they do not, and on CUDA 10.2 and later it additionally requires CUBLAS_WORKSPACE_CONFIG to be set to :4096:8 or :16:8 so cuBLAS uses a single workspace buffer. Setting torch.backends.cudnn.benchmark = False stops cuDNN from picking a different algorithm per run. All of that is worth doing and none of it constrains what happens when the shape changes. The flag guarantees that the same call twice gives the same answer. It says nothing about a different call that is mathematically the same.

Batch invariance is not configuration invariance

A kernel invariant to batch size can still change its answer when the tensor-parallel degree changes, because the contraction is then partitioned across devices differently and recombined in a different order. Training with FSDP on one topology and sampling with vLLM on another reintroduces exactly the mismatch batch invariance was meant to remove. Fixing it needs reduction trees whose shape is independent of the number of participating devices, which is what Tree-Based Invariant Kernels provide, reporting bit-wise identical results between vLLM and FSDP in an RL pipeline (Zhang et al., 2025, arXiv:2511.17826).

The performance cost is charged to everyone

Both implementations impose their overhead on the whole server, not on the requests that asked for determinism. Fixing the reduction strategy means the autotuner cannot pick a shape-optimal kernel for any request, so a workload where nobody cares about reproducibility still pays 24% to 55%. That asymmetry is the motivation for scheduling-based approaches, which decode on a fast nondeterministic path and enforce determinism through a verify-and-rollback loop, so cost scales with the fraction of traffic that needs the guarantee (Gond et al., 2026, LLM-42, arXiv:2601.17768).

Determinism is not correctness

A batch-invariant server reproduces its own answer exactly. It does not tell you the answer is right, and it does not detect a silently swapped or quantised model. Verification is a separate problem, and it can be attacked directly: DiFR fingerprints activations with random orthogonal projections and reports detecting 4-bit quantisation with AUC above 0.999 from as few as two output tokens (Karvonen et al., 2025, arXiv:2511.20621). Reproducibility and integrity are orthogonal properties and teams routinely conflate them.

Prefix caching reintroduces shape dependence

Reusing a cached prefix changes how much of the prompt is prefilled in this call, which changes the sequence lengths the attention kernel sees, which changes chunk boundaries unless the chunking is aligned to fixed offsets. SGLang's implementation had to align chunked-prefill truncation points explicitly for this reason. Any feature that varies how work is divided, prefix caching, chunked prefill, speculative decoding, disaggregated prefill and decode, is a potential source of shape variation that has to be handled individually.

Batch size 1 hides everything

The most common way this bug survives review is that it cannot be reproduced in the environment where people look for it. A single-request test on a developer machine is perfectly deterministic. The failure needs concurrency, and concurrency is exactly what the test environment lacks.

Alternative Designs

The distinction that matters when choosing is what you actually need. If you need the same answer twice, you need invariant kernels. If you need to know whether the answer came from the model you asked for, you need verification. If you need training and sampling to agree, you need invariance across whichever configurations differ between them, which is usually both batch size and parallelism degree.

How It Is Used in Practice

The clearest adoption case is on-policy reinforcement learning, and it is the one that turned this from a curiosity into engineering. Policy-gradient methods assume the logprobs used to weight the update are the logprobs under which the tokens were sampled. Sampling runs in an inference engine, training runs in a training framework, and the two use different kernels at different batch sizes on possibly different topologies. The discrepancy makes a nominally on-policy algorithm off-policy by an amount nobody measured.

The usual response is an importance-weighting correction, which works until it does not. With batch-invariant kernels the correction becomes unnecessary because the discrepancy is zero: the reported KL between sampler and trainer sits flat at zero, against roughly 0.001 with off-policy correction, while a run with no correction at all collapsed in reward around step 318. SGLang validated the same property end to end, reporting identical rollout responses and loss values for the first iterations of GRPO training on Qwen3-8B.

The second case is evaluation and regression testing. A benchmark that moves 0.4 points between runs of an unchanged model makes small genuine regressions undetectable, and teams compensate by running more seeds, which costs more than deterministic kernels would have.

The third is regulated and audited deployment, where "the model produced this output for this input" needs to be a reproducible claim rather than a probabilistic one.

Practically, the components are available. Thinking Machines released their batch-invariant kernels as a library, SGLang shipped deterministic modes across FlashInfer, FlashAttention 3, and Triton backends, and CUDA graphs recover a meaningful part of the overhead where they can be used. The remaining engineering is mostly integration: every scheduling feature that varies how work is split has to be made shape-stable, and that list grows with each new serving optimisation.

[IMAGE: Two training-run charts side by side over the same step axis. Left: KL divergence between sampler and trainer, one flat line at zero labelled "batch-invariant", one hovering near 0.001 labelled "off-policy correction", one drifting upward labelled "no correction". Right: reward for the same three runs, with the uncorrected run collapsing around step 318. Caption: "The reward collapse is downstream of a numerical disagreement nobody chose to introduce."]

[IMAGE: Bar chart of relative throughput for six configurations: vLLM default, vLLM deterministic unoptimised, vLLM deterministic optimised, SGLang FlashInfer deterministic, SGLang FA3 deterministic, SGLang Triton deterministic. Baseline normalised to 1.0. Caption: "Two independent implementations, one order of magnitude of agreement on what determinism costs."]

Insights Worth Remembering

1. Concurrency is not the culprit; shape selection is. A typical LLM forward pass contains no atomic adds, and the same kernel on the same batch is bit-identical. Blaming thread scheduling sends you looking in a place where the bug is not, which is why this went unexplained for as long as it did.

2. Batch invariance means each element's reduction order is fixed regardless of what else is in the batch. State it that way and the fix follows immediately: give up the autotuner's freedom in exactly the three reduction-heavy operations, matmul, RMSNorm, and attention.

3. Fixed split size, not fixed split count. This one detail separates deterministic attention from attention that looks deterministic. A fixed count makes chunk boundaries move as the KV cache grows, so a single request's reduction tree changes shape during its own generation.

4. The blast radius is asymmetric by design. A perturbation of \(10^{-6}\) in a logit changes nothing at all, until it changes an argmax, after which it changes every subsequent token. There is no proportionality between numerical error and output error once the output is discrete.

5. Determinism costs roughly a third, and everyone pays it. Two independent implementations landed between 24% and 112%, converging near 34% with tuned kernels. Because the cost comes from constraining the kernel, it applies to all traffic and not only to the requests that requested reproducibility.

6. Reproducibility and correctness are different properties. A batch-invariant server reliably reproduces a wrong answer. Detecting a swapped or quantised model needs verification, which is a separate mechanism with a separate cost.

7. This is a training-correctness bug wearing an infrastructure costume. The reason it got fixed in 2025 rather than 2019 is not that reproducibility became fashionable; it is that RL post-training made sampler-trainer logprob agreement a load-bearing assumption, and the assumption was false.

Open Questions

Can determinism be made close to free? Measured: constrained reduction orders cost 24% to 55% in the two published implementations, with CUDA graphs recovering 2.8x of the scheduling overhead. Unknown: how much of the residual gap is inherent to fixing the reduction order versus an artefact of kernels that have not yet been tuned under the constraint. The improvement from 55 s to 42 s in a single iteration suggests the ceiling has not been found.

Is verify-and-rollback the better trade at scale? Scheduling-based determinism promises cost proportional to the traffic requiring it, rather than a fixed tax on everything. Whether the rollback rate stays low under realistic mixed workloads, and what the resulting latency variance looks like at p99, has not been reported at production scale.

How many invariances are there? Batch size and tensor-parallel degree are two. Pipeline depth, expert routing in MoE models, speculative-decoding draft length, and prefix-cache hit patterns all plausibly change reduction structure. Nobody has published a systematic enumeration, so it is currently unclear whether "deterministic inference" is a finite checklist or an open-ended one.

Does non-invariance harm anything besides reproducibility and RL? It is plausible that shape-dependent numerics contribute to the reported variance in long-agent trajectories and in benchmark scores across serving configurations, since both involve many sequential argmax decisions. That is a hypothesis, not a measurement, and it would be straightforward to test by running an agent benchmark under invariant and non-invariant kernels at matched batch sizes.

Should reduction determinism be a hardware or compiler guarantee? Every current fix is a hand-written kernel decision. Whether a compiler could offer "reproducible reduction" as a compilation mode, with a declared cost, rather than leaving it to individual kernel authors, is an open design question in the ML compiler stack.

Sources and Further Reading

1. He, H., and Thinking Machines Lab (2025). "Defeating Nondeterminism in LLM Inference." Published 10 September 2025. thinkingmachines.ai
2. LMSYS Org / SGLang Team (2025). "Towards Deterministic Inference in SGLang and Reproducible RL Training." Published 22 September 2025. lmsys.org
3. Zhang, Z., Ding, X., Yuan, J., Liu, R., Mao, H., Xing, J., & Liu, Z. (2025). "Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch." arXiv:2511.17826
4. Gond, R., Kamath, A. K., Ramjee, R., & Panwar, A. (2026). "LLM-42: Enabling Determinism in LLM Inference with Verified Speculation." arXiv:2601.17768
5. Karvonen, A., Reuter, D., Rinberg, R., Marks, L., Garriga-Alonso, A., & Warr, K. (2025). "DiFR: Inference Verification Despite Nondeterminism." arXiv:2511.20621
6. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP 2023. arXiv:2309.06180
7. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., & Chun, B.-G. (2022). "Orca: A Distributed Serving System for Transformer-Based Generative Models." OSDI 2022. USENIX
8. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." NeurIPS 2022. arXiv:2205.14135
9. Zhang, B., & Sennrich, R. (2019). "Root Mean Square Layer Normalization." NeurIPS 2019. arXiv:1910.07467
10. PyTorch documentation. "Reproducibility." docs.pytorch.org
11. Goldberg, D. (1991). "What Every Computer Scientist Should Know About Floating-Point Arithmetic." ACM Computing Surveys, 23(1), 5-48.
12. IEEE (2019). "IEEE Standard for Floating-Point Arithmetic." IEEE Std 754-2019.

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.