Tagged “pretraining”

3 posts.

All Model Architecture19 Training & Alignment22 Inference & Serving20 Agents & Orchestration11 Reasoning & Evaluation27 Safety, Security & Governance7 Platforms & Practice22

Training & Alignment 24 min

At Sixteen Thousand GPUs, Something Is Always Broken: Failures, Stragglers and Silent Data Corruption in Training Clusters

Over 54 days of Llama 3 405B pre-training, the job was interrupted 466 times, roughly once every three hours. At that failure rate the checkpoint interval barely matters; what decides how much of a sixteen-thousand-GPU cluster do…

gpu-fleet-and-capacity distributed-training reliability gpu ∑ ◫

Platforms & Practice 25 min

Learning Without Labels: The Collapse Problem at the Heart of Self-Supervised Learning

Every joint-embedding method has the same trivial solution available to it: map every input to the same vector. The history of self-supervised learning is the history of preventing that, and the methods that look most different f…

self-supervised-learning contrastive-learning representation-learning embeddings ∑ ◫

Training & Alignment 24 min

The Bandwidth Wall: How Low-Communication Training Unbundled the Datacentre

Data-parallel training all-reduces the entire gradient after every step, which is why frontier pretraining happens inside one building with a purpose-built fabric. DiLoCo synchronises every five hundred steps instead of every one…

distributed-training scaling infrastructure optimisation ∑ ◫