Tagged “scaling”

Model Architecture 25 min

Mixture of Experts: How Frontier LLMs Achieve Scale Without Proportional Cost

Every frontier LLM released since 2023 uses some form of Mixture of Experts. The idea is deceptively simple: build a model with hundreds of billions of parameters but activate only a fraction for each token. The result is dense-m…

moe architecture scaling inference ∑ ◫

Training & Alignment 21 min

Muon and MuonClip: The Optimizer That Broke Adam's Monopoly on LLM Pretraining

For eight years Adam owned every serious pretraining run. Then a competitive-speedrun optimizer that orthogonalizes its own gradient updates scaled from a 124M-parameter toy to a trillion-parameter model with zero loss spikes. He…

optimization llm-training muon adamw ∑ ◫

Model Architecture 22 min

Patches Over Tokens: How the Byte Latent Transformer Kills the Tokenizer

A tokenizer decides in advance how many bits of compute every piece of text deserves. The Byte Latent Transformer throws that decision out and lets the entropy of the raw bytes allocate compute instead, matching Llama 3 at 8B par…

byte-latent-transformer tokenization llm-architecture dynamic-patching ∑ ◫

Reasoning & Evaluation 18 min

Test-Time Compute: How Reasoning Models Buy Intelligence by the Token

A 7B model that thinks for thirty seconds can outscore a 70B model that answers instantly. Test-time compute is the lever that makes this possible, and it is quietly rewriting the economics of model design.

reasoning test-time-compute inference reinforcement-learning ∑ ◫

Training & Alignment 9 min

The $8M trillion: when frontier-grade training falls out of the lab

DeepSeek shipped a 1.6T-parameter model in April 2026 with native Ascend inference support while OpenAI committed $500B to Stargate. Both can be right, but only one of them is a moat.

training compute scaling economics ∑

Training & Alignment 24 min

The Bandwidth Wall: How Low-Communication Training Unbundled the Datacentre

Data-parallel training all-reduces the entire gradient after every step, which is why frontier pretraining happens inside one building with a purpose-built fabric. DiLoCo synchronises every five hundred steps instead of every one…

distributed-training scaling infrastructure optimisation ∑ ◫

Training & Alignment 26 min

The Bias-Variance Tradeoff Is a Special Case: Double Descent, Benign Overfitting, and Grokking

A network that fits ImageNet with randomly shuffled labels should not generalise on real ones. It does. That single experiment invalidated the textbook account of why machine learning works, and the three phenomena that replaced …

training-dynamics generalisation double-descent grokking ∑ ◫

Model Architecture 4 min

The Length of a Thought: Why Context Windows Became the New Battleground

A 512-token model could read a paragraph. A million-token model can read a novel. The journey between those two numbers is a story about quadratic cost, clever approximations, and what it means for a machine to "remember."

transformers attention context scaling

Model Architecture 32 min

The Wavelength Problem: How One Number Decides a Model's Context Length

Llama 2 and Llama 3 share an attention implementation almost line for line. One constant differs: the RoPE base went from 10,000 to 500,000. That number sets the entire wavelength spectrum of the model's position signal, and it h…

positional-encoding rope long-context transformers ∑ ◫