Tagged “training-dynamics”

4 posts.

All Model Architecture21 Training & Alignment22 Inference & Serving23 Agents & Orchestration12 Reasoning & Evaluation27 Safety, Security & Governance9 Platforms & Practice22

Training & Alignment 27 min

Model Merging: Why Averaging Weights Works, and Where the Free Lunch Ends

Three 7B models that each scored under 30% on Japanese maths were averaged into one that scored 52%. No gradient was computed. Weight-space arithmetic is the cheapest capability gain in the field and the easiest one to fool yours…

model-merging fine-tuning task-vectors peft ∑ ◫

Training & Alignment 26 min

The Bias-Variance Tradeoff Is a Special Case: Double Descent, Benign Overfitting, and Grokking

A network that fits ImageNet with randomly shuffled labels should not generalise on real ones. It does. That single experiment invalidated the textbook account of why machine learning works, and the three phenomena that replaced …

training-dynamics generalisation double-descent grokking ∑ ◫

Model Architecture 33 min

The Training Run That Never Ends: Continued Pretraining, Mid-Training and Model Growth

Forty million tokens, a rounding error against Llama 3's fifteen trillion, moved GSM8K by 24 percent. That is not a pretraining result; it is a result about what happens after pretraining is nominally over. A pretrained checkpoin…

pretraining continual-learning scaling training-dynamics ∑ ◫

Model Architecture 24 min

Why GANs Lost Image Generation, and Why Adversarial Losses Are Everywhere Anyway

In 2021 a diffusion model beat BigGAN-deep on ImageNet while nearly doubling its recall, and GANs stopped being the default way to generate images. Yet latent autoencoders, SDXL-Turbo, HiFi-GAN and the neural audio codecs behind …

adversarial-generative-models gans image-generation diffusion-models ∑ ◫