Tagged “scaling”

6 posts.

All Model Architecture13 Training & Alignment8 Inference & Serving11 Agents & Orchestration10 Reasoning & Evaluation7 Safety, Security & Governance3 Platforms & Practice11

Model Architecture 25 min

Mixture of Experts: How Frontier LLMs Achieve Scale Without Proportional Cost

Every frontier LLM released since 2023 uses some form of Mixture of Experts. The idea is deceptively simple: build a model with hundreds of billions of parameters but activate only a fraction for each token. The result is dense-m…

moe architecture scaling inference ∑ ◫

Training & Alignment 21 min

Muon and MuonClip: The Optimizer That Broke Adam's Monopoly on LLM Pretraining

For eight years Adam owned every serious pretraining run. Then a competitive-speedrun optimizer that orthogonalizes its own gradient updates scaled from a 124M-parameter toy to a trillion-parameter model with zero loss spikes. He…

optimization llm-training muon adamw ∑ ◫

Model Architecture 22 min

Patches Over Tokens: How the Byte Latent Transformer Kills the Tokenizer

A tokenizer decides in advance how many bits of compute every piece of text deserves. The Byte Latent Transformer throws that decision out and lets the entropy of the raw bytes allocate compute instead, matching Llama 3 at 8B par…

byte-latent-transformer tokenization llm-architecture dynamic-patching ∑ ◫

Reasoning & Evaluation 18 min

Test-Time Compute: How Reasoning Models Buy Intelligence by the Token

A 7B model that thinks for thirty seconds can outscore a 70B model that answers instantly. Test-time compute is the lever that makes this possible, and it is quietly rewriting the economics of model design.

reasoning test-time-compute inference reinforcement-learning ∑ ◫

Training & Alignment 9 min

The $8M trillion: when frontier-grade training falls out of the lab

DeepSeek shipped a 1.6T-parameter model in April 2026 with native Ascend inference support while OpenAI committed $500B to Stargate. Both can be right, but only one of them is a moat.

training compute scaling economics ∑

Model Architecture 4 min

The Length of a Thought: Why Context Windows Became the New Battleground

A 512-token model could read a paragraph. A million-token model can read a novel. The journey between those two numbers is a story about quadratic cost, clever approximations, and what it means for a machine to "remember."

transformers attention context scaling