Model Architecture

How frontier models are built: attention variants, sparsity, memory and state.

All Model Architecture19 Training & Alignment22 Inference & Serving18 Agents & Orchestration11 Reasoning & Evaluation26 Safety, Security & Governance7 Platforms & Practice20

Model Architecture 4 min

The Length of a Thought: Why Context Windows Became the New Battleground

A 512-token model could read a paragraph. A million-token model can read a novel. The journey between those two numbers is a story about quadratic cost, clever approximations, and what it means for a machine to "remember."

transformers attention context scaling

Model Architecture 23 min

The Residual Stream: The Transformer's Shared Memory Bus

Stop reading the transformer as a pipeline of 96 layers, each transforming the output of the last. Read it as one shared communication channel that every attention head and MLP merely edits. This one reframing, formalised by inte…

transformer-anatomy interpretability residual-stream attention ∑ ◫

Model Architecture 24 min

The Score Is All You Need: How Energy-Based Models, Langevin Dynamics and Diffusion Became One Theory

For decades the normalising constant made energy-based models nearly impossible to train at scale. Between 2019 and 2022 the field stopped computing it and learned its gradient instead, and score matching, Langevin sampling and d…

energy-based-and-score-models diffusion-models diffusion generative-ai ∑ ◫

Model Architecture 32 min

The Wavelength Problem: How One Number Decides a Model's Context Length

Llama 2 and Llama 3 share an attention implementation almost line for line. One constant differs: the RoPE base went from 10,000 to 500,000. That number sets the entire wavelength spectrum of the model's position signal, and it h…

positional-encoding rope long-context transformers ∑ ◫

Model Architecture 26 min

Titans: The Sequence Architecture That Learns to Remember While It Runs

Most long-context models compress the past into a fixed-size vector and hope nothing important got squeezed out. Titans instead gives the model a small neural network as its memory, and lets that network keep training on the data…

architecture long-context memory test-time-training ∑ ◫

Model Architecture 18 min

Trainable Sparse Attention: When the Model Learns What to Skip

For years, sparse attention was a bolt-on applied after training to shrink the KV cache. NSA, MoBA, and DeepSeek's DSA flip that order, training the sparsity end to end and reporting up to 11x faster decoding without losing accuracy.

attention long-context sparse-attention transformers ∑ ◫

Model Architecture 24 min

Why GANs Lost Image Generation, and Why Adversarial Losses Are Everywhere Anyway

In 2021 a diffusion model beat BigGAN-deep on ImageNet while nearly doubling its recall, and GANs stopped being the default way to generate images. Yet latent autoencoders, SDXL-Turbo, HiFi-GAN and the neural audio codecs behind …

adversarial-generative-models gans image-generation diffusion-models ∑ ◫

Browse by topic