Inference & Serving
24 min
Your Answer Depends on Who Else Is in the Batch
A thousand identical requests to the same model at temperature zero produced eighty different completions. The cause is not random seeds, not sampling, and not GPU atomics. It is that reduction kernels change their summation orde…