State Space Models
S4, Mamba, selective scan, the recurrence-convolution duality, and where linear-time sequence models pay off.
11concepts
94flashcards
82minutes of reading
- 01 Hybrid Attention-SSM Architectures Why the strongest linear-time models are not pure state space models, what a small number of full attention layers restores, and how to reason about the mixing ratio.
- 02 The Fixed State Budget at Inference Why a state space model decodes in constant memory per sequence, how to do the arithmetic against a KV cache, and the operational consequences that constant memory buys and costs.