State Space Models
S4, Mamba, selective scan, the recurrence-convolution duality, and where linear-time sequence models pay off.
5concepts
58flashcards
36minutes of reading
- 01 Hybrid Attention-SSM Architectures Why the strongest linear-time models are not pure state space models, what a small number of full attention layers restores, and how to reason about the mixing ratio.
- 02 HiPPO and Structured State Initialisation Why a randomly initialised state space layer fails on long sequences, what optimal polynomial projection of history gives you instead, and how the structure that makes it work also makes it computable.
- 03 Parallel Scan and Hardware-Aware SSM Kernels How an associative scan recovers training parallelism after selectivity destroys the convolution, and why the arithmetic-intensity argument means the kernel is the architecture.
- 04 Selectivity and Input-Dependent Parameters What linear time-invariant models fundamentally cannot do, how making the state space parameters functions of the input fixes it, and the computational bill that change immediately creates.
- 05 The Recurrence-Convolution Duality Why a linear state space layer can be run as a parallel convolution during training and as a constant-memory recurrence at inference, and what the word "linear" is buying.