All writing

124 long-form pieces on architecture, training, inference, agents and safety.

All Model Architecture19 Training & Alignment22 Inference & Serving18 Agents & Orchestration11 Reasoning & Evaluation26 Safety, Security & Governance7 Platforms & Practice21

Reasoning & Evaluation 23 min

The Progress Illusion in Recommender Systems: Weak Baselines, Sampled Metrics and Leaky Splits

In 2019 a careful team could reproduce only 7 of 18 neural recommenders from top venues, and 6 of those 7 lost to nearest-neighbour heuristics. The models were not the problem. The protocol was: untuned baselines, metrics compute…

recommender-systems evaluation metrics benchmarks ∑ ◫

Platforms & Practice 6 min

The Quiet Standard: How the Model Context Protocol Became the USB-C of AI

The most consequential AI release of late 2024 was not a model. It was a protocol. The story of how a single open standard ended the N-times-M integration nightmare is the story of every standard that ever mattered.

mcp standards interoperability agents

Model Architecture 23 min

The Residual Stream: The Transformer's Shared Memory Bus

Stop reading the transformer as a pipeline of 96 layers, each transforming the output of the last. Read it as one shared communication channel that every attention head and MLP merely edits. This one reframing, formalised by inte…

transformer-anatomy interpretability residual-stream attention ∑ ◫

Model Architecture 24 min

The Score Is All You Need: How Energy-Based Models, Langevin Dynamics and Diffusion Became One Theory

For decades the normalising constant made energy-based models nearly impossible to train at scale. Between 2019 and 2022 the field stopped computing it and learned its gradient instead, and score matching, Langevin sampling and d…

energy-based-and-score-models diffusion-models diffusion generative-ai ∑ ◫

Model Architecture 32 min

The Wavelength Problem: How One Number Decides a Model's Context Length

Llama 2 and Llama 3 share an attention implementation almost line for line. One constant differs: the RoPE base went from 10,000 to 500,000. That number sets the entire wavelength spectrum of the model's position signal, and it h…

positional-encoding rope long-context transformers ∑ ◫

Reasoning & Evaluation 13 min

The reasoning-model bubble: when test-time compute stops paying

o3, R1 and Claude extended thinking are a real capability shift on a narrow slice of tasks. They are also being shoved into product surfaces that punish every property reasoning models exhibit - and the bill is starting to arrive.

reasoning test-time-compute economics evaluation ∑

Model Architecture 26 min

Titans: The Sequence Architecture That Learns to Remember While It Runs

Most long-context models compress the past into a fixed-size vector and hope nothing important got squeezed out. Titans instead gives the model a small neural network as its memory, and lets that network keep training on the data…

architecture long-context memory test-time-training ∑ ◫

Model Architecture 18 min

Trainable Sparse Attention: When the Model Learns What to Skip

For years, sparse attention was a bolt-on applied after training to shrink the KV cache. NSA, MoBA, and DeepSeek's DSA flip that order, training the sparsity end to end and reporting up to 11x faster decoding without losing accuracy.

attention long-context sparse-attention transformers ∑ ◫

Reasoning & Evaluation 24 min

Twenty Tests, One False Discovery: Multiple Testing From Bonferroni to the False Discovery Rate

A dead Atlantic salmon, scanned in 2009, showed 16 'active' voxels at p below 0.001; every procedure that controlled an error rate across the family found none. This is the argument over what that error rate should be, from Holm …

statistical-inference statistics experimentation ab-testing ∑ ◫

Training & Alignment 24 min

Vision-Language-Action Models: The Action Interface Is the Hard Part

A language model eats trillions of tokens scraped for free. The largest open robot dataset is 527 skills gathered by hand across 21 institutions. That asymmetry, not model capacity, is what makes robot learning hard, and it expla…

robotics-embodied vla imitation-learning flow-matching ∑ ◫

Platforms & Practice 24 min

What Structure Buys You: Knowledge Graphs in the Age of Language Models

In 2019 a masked language model recalled facts almost as well as a relation extractor with an oracle entity linker, and people asked whether knowledge graphs were finished. They were not, but the reason is narrower than their adv…

knowledge-graphs rag retrieval hybrid-retrieval ∑ ◫