Efficient Architectures
Small language models, depth-width tradeoffs, weight sharing, and architectures designed for a latency budget.
11concepts
97flashcards
81minutes of reading
- 01 Architecting to a Latency Budget How to design a model backwards from a millisecond target using arithmetic intensity, why parameter count is the wrong currency, and the design moves that actually reduce time to first and subsequent tokens.
- 02 Cross-Layer KV Sharing Why sharing key and value tensors between layers is the one KV cache reduction that composes with GQA and sliding windows, how CLA and YOCO differ in what they let you skip, and the prefill and quality costs that come with each layout.
- 03 Depth Versus Width Tradeoffs Why two models with identical parameter counts behave differently depending on how the parameters are arranged, what depth buys that width cannot, and how the hardware votes for width.
- 04 Hardware-Aware Architecture Search Why optimising a model for FLOPs produces slow models, how measured latency became the objective instead, and what makes once-for-all supernet training the practical form of the idea.
- 05 Looped Transformers and Recurrent Depth What you get when the same transformer block runs several times instead of stacking distinct layers, why depth rather than parameter count is what many reasoning problems need, and the memory-bandwidth and interpretability costs of buying compute this way.
- 06 Nested and Elastic Transformers How training one model with deliberately nested sub-networks lets you extract a whole family of deployable models at no extra training cost, what Mix'n'Match buys over a fixed ladder of sizes, and why the parent still has to be resident in memory for the child to run.