Batch Invariance and Numerical Reproducibility

Because the logits differ before sampling. Thinking Machines sampled Qwen3-235B-A22B 1,000 times at temperature 0 and got 80 distinct completions, first diverging at token 103. Greedy decoding faithfully picks the argmax; the argmax itself flipped because the last bits of two near-tied logits changed between runs. Nondeterminism lives in the forward pass, not the sampler.

Because a typical LLM forward pass contains no atomic adds. Run the same kernel on the same batch twice and it is bit-identical. The real dependency is on shape: kernels select a parallelisation and therefore a reduction order from the input dimensions, so the same matmul reduces in a different order at batch 1 than at batch 32. Floating-point addition is not associative, so the low-order bits differ.

That each element's reduction order is fixed regardless of batch size or position in the batch. A continuously-batching server groups your request with whatever unrelated traffic arrived at the same moment, so without batch invariance your logits are a function of other users' requests. Nothing about your input changed; the summation order did, and one flipped comparison sends the whole continuation elsewhere.

A fixed split count makes chunk size a function of KV length, so the reduction tree changes shape as the cache grows during decoding, and the same token accumulates differently at step 100 and step 1,000. Fixed-size chunks keep each element's accumulation path constant: a 1,000-element cache becomes three 256-element chunks plus one of 232, and the arrangement is unaffected by how many query tokens are in flight.

Thinking Machines report 55 seconds for an unoptimised deterministic vLLM against 26 for the default on a 1,000-sequence Qwen3-8B run, improving to 42 with a better attention kernel. SGLang's independent implementation across FlashInfer, FlashAttention 3, and Triton reports an average 34.35% slowdown, spanning 24.4% to 55.1%. The cost is giving up the autotuner's freedom to pick a shape-optimal reduction.

Because the sampler and the trainer must agree on logprobs for the policy gradient to be on-policy. If their kernels reduce differently, training is quietly off-policy and depends on an importance-weighting correction to stay stable. With batch-invariant kernels the KL between sampler and trainer sits flat at zero, versus roughly 0.001 under off-policy correction; a run with no correction at all collapsed in reward around step 318.

That flag buys run-to-run determinism at a fixed shape: it selects deterministic implementations where they exist, errors where they do not, and on CUDA 10.2 and later also needs CUBLAS_WORKSPACE_CONFIG set to :4096:8 or :16:8. None of it constrains how a kernel reduces when the shape changes. Batch invariance is a different property, and tensor-parallel-size invariance is a third one again.

Because the logits differ before sampling. Thinking Machines sampled Qwen3-235B-A22B 1,000 times at temperature 0 and got 80 distinct completions, first diverging at token 103. Greedy decoding faithfully picks the argmax; the argmax itself flipped because the last bits of two near-tied logits changed between runs. Nondeterminism lives in the forward pass, not the sampler.

Because a typical LLM forward pass contains no atomic adds. Run the same kernel on the same batch twice and it is bit-identical. The real dependency is on shape: kernels select a parallelisation and therefore a reduction order from the input dimensions, so the same matmul reduces in a different order at batch 1 than at batch 32. Floating-point addition is not associative, so the low-order bits differ.

That each element's reduction order is fixed regardless of batch size or position in the batch. A continuously-batching server groups your request with whatever unrelated traffic arrived at the same moment, so without batch invariance your logits are a function of other users' requests. Nothing about your input changed; the summation order did, and one flipped comparison sends the whole continuation elsewhere.

A fixed split count makes chunk size a function of KV length, so the reduction tree changes shape as the cache grows during decoding, and the same token accumulates differently at step 100 and step 1,000. Fixed-size chunks keep each element's accumulation path constant: a 1,000-element cache becomes three 256-element chunks plus one of 232, and the arrangement is unaffected by how many query tokens are in flight.

Thinking Machines report 55 seconds for an unoptimised deterministic vLLM against 26 for the default on a 1,000-sequence Qwen3-8B run, improving to 42 with a better attention kernel. SGLang's independent implementation across FlashInfer, FlashAttention 3, and Triton reports an average 34.35% slowdown, spanning 24.4% to 55.1%. The cost is giving up the autotuner's freedom to pick a shape-optimal reduction.

Because the sampler and the trainer must agree on logprobs for the policy gradient to be on-policy. If their kernels reduce differently, training is quietly off-policy and depends on an importance-weighting correction to stay stable. With batch-invariant kernels the KL between sampler and trainer sits flat at zero, versus roughly 0.001 under off-policy correction; a run with no correction at all collapsed in reward around step 318.

That flag buys run-to-run determinism at a fixed shape: it selects deterministic implementations where they exist, errors where they do not, and on CUDA 10.2 and later also needs CUBLAS_WORKSPACE_CONFIG set to :4096:8 or :16:8. None of it constrains how a kernel reduces when the shape changes. Batch invariance is a different property, and tensor-parallel-size invariance is a third one again.