Training & Alignment
24 min
At Sixteen Thousand GPUs, Something Is Always Broken: Failures, Stragglers and Silent Data Corruption in Training Clusters
Over 54 days of Llama 3 405B pre-training, the job was interrupted 466 times, roughly once every three hours. At that failure rate the checkpoint interval barely matters; what decides how much of a sixteen-thousand-GPU cluster do…