Efficient Architectures
Small language models, depth-width tradeoffs, weight sharing, and architectures designed for a latency budget.
11concepts
97flashcards
81minutes of reading
- 01 Depthwise Separable Convolutions and Mobile Networks Splitting a convolution into a per-channel spatial filter and a 1x1 channel mixer cuts its arithmetic by nearly an order of magnitude, which built MobileNet and EfficientNet, and also produced the field's clearest case of FLOPs failing to predict speed.
- 02 Per-Layer Embeddings and Parameter Offloading Why a model's parameter count stopped predicting its accelerator memory footprint, how per-layer embeddings move a large block of weights to cheap memory that only needs a lookup, and what the PCIe or unified-memory path costs when the trick is applied to the wrong tensors.
- 03 Small Language Models and the Overtraining Regime Why compute-optimal training is the wrong objective when inference dominates the bill, how far past Chinchilla the good small models actually go, and what stops the trend.
- 04 The Hardware Lottery in Architecture Choice Why architectural ideas succeed or fail partly on their fit to whatever silicon and kernels happen to exist, with the concrete cases (unstructured sparsity, state space models, capsule networks) where the fit rather than the idea decided the outcome.
- 05 Weight Tying and Parameter Sharing Where reusing one set of weights in several places is nearly free, where it costs real capability, and why the embedding matrix is the case everyone gets right and the layer stack is the case everyone gets wrong.