Model Architecture

How frontier models are built: attention variants, sparsity, memory and state.

All Model Architecture19 Training & Alignment22 Inference & Serving18 Agents & Orchestration11 Reasoning & Evaluation26 Safety, Security & Governance7 Platforms & Practice20

Model Architecture 23 min

Context Rot: Why Bigger Context Windows Don't Mean Better Retrieval

A million-token window promises perfect recall of everything you feed it. Controlled tests on 18 frontier models show recall degrading steadily, unevenly, and well before the window fills, a pattern researchers now call context rot.

llm long-context rag context-engineering ∑ ◫

Model Architecture 20 min

Do Transformers Need Three Projections? Rethinking Q, K, and V

Every attention head learns three weight matrices for query, key, and value. A 2026 study trained models up to 1.2B parameters to ask which of them are actually load-bearing, and found that keys and values can share one projectio…

transformers attention kv-cache inference ∑ ◫

Model Architecture 21 min

From 4K to a Million Tokens: How RoPE Scaling, YaRN, and Ring Attention Stretch the Context Window

A model trained on 4,096 tokens can be coaxed into reading a quarter-million without retraining from scratch. The trick is not bigger attention; it is lying to the model about position, and splitting the sequence across a ring of GPUs.

long-context rope yarn ring-attention ∑ ◫

Model Architecture 42 min

From Nano Banana to Vision Banana: How Google DeepMind Turned an Image Generator into a Generalist Vision System

Training a model to generate photorealistic images teaches it geometry, semantics, depth, and object relationships. Google DeepMind's Vision Banana proves that a lightweight instruction-tuning pass over an image generator can bea…

nano-banana vision-banana image-generation computer-vision ∑ ◫

Model Architecture 24 min

From Normalising Flows to Flow Matching: How the Change-of-Variables Idea Survived Its Own Architecture

For five years, normalising flows bent every layer around one number: the log-determinant of a Jacobian. Flow matching kept the invertible transport map, stopped computing that number during training, and ended up inside Stable D…

variational-and-flow-models flow-matching diffusion-models generative-ai ∑ ◫

Model Architecture 20 min

Half Mamba, Half Attention: Why Hybrid State-Space Models Took Over

Pure Mamba was supposed to replace attention. Instead the most efficient open models in 2025 are roughly seven-eighths Mamba and one-eighth attention. The reason is the KV cache, and what each layer can and cannot remember.

state-space-models mamba transformers long-context ∑ ◫

Model Architecture 25 min

Learned Weather Forecasting: What Neural Networks Replaced, and What They Did Not

A 36.7-million-parameter graph network produces a 10-day global forecast in under a minute and beats the world's best physics-based system on 90 percent of verification targets. It also cannot start without that system's data ass…

ai-for-science weather diffusion graph-neural-networks ∑ ◫

Model Architecture 25 min

Mixture of Experts: How Frontier LLMs Achieve Scale Without Proportional Cost

Every frontier LLM released since 2023 uses some form of Mixture of Experts. The idea is deceptively simple: build a model with hundreds of billions of parameters but activate only a fraction for each token. The result is dense-m…

moe architecture scaling inference ∑ ◫

Model Architecture 22 min

Multi-Head Latent Attention: How DeepSeek Compressed the KV Cache Without Losing Quality

Every token an LLM generates forces it to reload the keys and values of every token that came before. Multi-Head Latent Attention rewrites that trade by caching one compressed vector instead of dozens of separate heads, cutting m…

llm-architecture attention-mechanisms deepseek kv-cache ∑ ◫

Model Architecture 15 min

Orthogonalizing Memory Reads: A Muon Trick for Noisy Recurrent Recall

A short experimental note claims that orthogonalizing an mLSTM's matrix memory at read time, borrowing Muon's Newton-Schulz step, sharpens noisy associative recall exactly where the baseline is failing. We walk through the mechan…

recurrent-models mlstm xlstm associative-recall ∑ ◫

Model Architecture 22 min

Patches Over Tokens: How the Byte Latent Transformer Kills the Tokenizer

A tokenizer decides in advance how many bits of compute every piece of text deserves. The Byte Latent Transformer throws that decision out and lets the entropy of the raw bytes allocate compute instead, matching Llama 3 at 8B par…

byte-latent-transformer tokenization llm-architecture dynamic-patching ∑ ◫