Tagged “moe”

2 posts.

All Model Architecture19 Training & Alignment22 Inference & Serving18 Agents & Orchestration11 Reasoning & Evaluation26 Safety, Security & Governance7 Platforms & Practice20

Model Architecture 25 min

Mixture of Experts: How Frontier LLMs Achieve Scale Without Proportional Cost

Every frontier LLM released since 2023 uses some form of Mixture of Experts. The idea is deceptively simple: build a model with hundreds of billions of parameters but activate only a fraction for each token. The result is dense-m…

moe architecture scaling inference ∑ ◫

Training & Alignment 21 min

Muon and MuonClip: The Optimizer That Broke Adam's Monopoly on LLM Pretraining

For eight years Adam owned every serious pretraining run. Then a competitive-speedrun optimizer that orthogonalizes its own gradient updates scaled from a 124M-parameter toy to a trillion-parameter model with zero loss spikes. He…

optimization llm-training muon adamw ∑ ◫