Tagged “distributed-training”

3 posts.

All Model Architecture23 Training & Alignment23 Inference & Serving24 Agents & Orchestration13 Reasoning & Evaluation28 Safety, Security & Governance9 Platforms & Practice22

Training & Alignment 24 min

At Sixteen Thousand GPUs, Something Is Always Broken: Failures, Stragglers and Silent Data Corruption in Training Clusters

Over 54 days of Llama 3 405B pre-training, the job was interrupted 466 times, roughly once every three hours. At that failure rate the checkpoint interval barely matters; what decides how much of a sixteen-thousand-GPU cluster do…

gpu-fleet-and-capacity distributed-training reliability gpu ∑ ◫

Training & Alignment 30 min

Feeding the Accelerator: Why the Training Loop Is a Storage Problem

A 75% idle GPU is usually not a compute problem. Across the configurations one VLDB study measured, DNN training spent between 10% and 70% of every epoch blocked on I/O, and the storage system was not busy. The data path has its …

training storage data-loading infrastructure ∑ ◫

Training & Alignment 24 min

The Bandwidth Wall: How Low-Communication Training Unbundled the Datacentre

Data-parallel training all-reduces the entire gradient after every step, which is why frontier pretraining happens inside one building with a purpose-built fabric. DiLoCo synchronises every five hundred steps instead of every one…

distributed-training scaling infrastructure optimisation ∑ ◫