Efficient Architectures
Small language models, depth-width tradeoffs, weight sharing, and architectures designed for a latency budget.
11concepts
97flashcards
81minutes of reading
- 01 Depthwise Separable Convolutions and Mobile Networks Splitting a convolution into a per-channel spatial filter and a 1x1 channel mixer cuts its arithmetic by nearly an order of magnitude, which built MobileNet and EfficientNet, and also produced the field's clearest case of FLOPs failing to predict speed.
- 02 Per-Layer Embeddings and Parameter Offloading Why a model's parameter count stopped predicting its accelerator memory footprint, how per-layer embeddings move a large block of weights to cheap memory that only needs a lookup, and what the PCIe or unified-memory path costs when the trick is applied to the wrong tensors.
- 03 Small Language Models and the Overtraining Regime Why compute-optimal training is the wrong objective when inference dominates the bill, how far past Chinchilla the good small models actually go, and what stops the trend.
- 04 The Hardware Lottery in Architecture Choice Why architectural ideas succeed or fail partly on their fit to whatever silicon and kernels happen to exist, with the concrete cases (unstructured sparsity, state space models, capsule networks) where the fit rather than the idea decided the outcome.
- 05 Weight Tying and Parameter Sharing Where reusing one set of weights in several places is nearly free, where it costs real capability, and why the embedding matrix is the case everyone gets right and the layer stack is the case everyone gets wrong.
- 06 Architecting to a Latency Budget How to design a model backwards from a millisecond target using arithmetic intensity, why parameter count is the wrong currency, and the design moves that actually reduce time to first and subsequent tokens.
- 07 Cross-Layer KV Sharing Why sharing key and value tensors between layers is the one KV cache reduction that composes with GQA and sliding windows, how CLA and YOCO differ in what they let you skip, and the prefill and quality costs that come with each layout.
- 08 Depth Versus Width Tradeoffs Why two models with identical parameter counts behave differently depending on how the parameters are arranged, what depth buys that width cannot, and how the hardware votes for width.
- 09 Hardware-Aware Architecture Search Why optimising a model for FLOPs produces slow models, how measured latency became the objective instead, and what makes once-for-all supernet training the practical form of the idea.
- 10 Looped Transformers and Recurrent Depth What you get when the same transformer block runs several times instead of stacking distinct layers, why depth rather than parameter count is what many reasoning problems need, and the memory-bandwidth and interpretability costs of buying compute this way.
- 11 Nested and Elastic Transformers How training one model with deliberately nested sub-networks lets you extract a whole family of deployable models at no extra training cost, what Mix'n'Match buys over a fixed ladder of sizes, and why the parent still has to be resident in memory for the child to run.