Efficient Architectures
Small language models, depth-width tradeoffs, weight sharing, and architectures designed for a latency budget.
5concepts
58flashcards
36minutes of reading
- 01 Small Language Models and the Overtraining Regime Why compute-optimal training is the wrong objective when inference dominates the bill, how far past Chinchilla the good small models actually go, and what stops the trend.
- 02 Weight Tying and Parameter Sharing Where reusing one set of weights in several places is nearly free, where it costs real capability, and why the embedding matrix is the case everyone gets right and the layer stack is the case everyone gets wrong.