State Space Models
S4, Mamba, selective scan, the recurrence-convolution duality, and where linear-time sequence models pay off.
11concepts
94flashcards
82minutes of reading
- 01 Hybrid Attention-SSM Architectures Why the strongest linear-time models are not pure state space models, what a small number of full attention layers restores, and how to reason about the mixing ratio.
- 02 The Fixed State Budget at Inference Why a state space model decodes in constant memory per sequence, how to do the arithmetic against a KV cache, and the operational consequences that constant memory buys and costs.
- 03 Discretising Continuous State Spaces How a continuous-time state space layer becomes a recurrence you can run on tokens, why zero-order hold and the bilinear transform disagree at large step sizes, and why the step size ends up acting as a learned timescale and a gate.
- 04 Distilling Transformers into Linear-Time Models How a pretrained transformer's attention projections are reused to initialise state space layers, why the conversion is staged rather than end to end, and what the resulting hybrids do and do not recover.
- 05 HiPPO and Structured State Initialisation Why a randomly initialised state space layer fails on long sequences, what optimal polynomial projection of history gives you instead, and how the structure that makes it work also makes it computable.
- 06 Long Convolution Models and Implicit Filters How H3 and Hyena reached subquadratic sequence mixing through FFT convolutions with implicitly parameterised filters, what data-controlled gating contributed, and why selectivity displaced the whole family.
- 07 Parallel Scan and Hardware-Aware SSM Kernels How an associative scan recovers training parallelism after selectivity destroys the convolution, and why the arithmetic-intensity argument means the kernel is the architecture.
- 08 Selectivity and Input-Dependent Parameters What linear time-invariant models fundamentally cannot do, how making the state space parameters functions of the input fixes it, and the computational bill that change immediately creates.
- 09 State Capacity and Associative Recall The information-theoretic reason a fixed-size recurrent state cannot copy arbitrary context, what MQAR measures that perplexity hides, and why the gap shows up on exactly the tasks production systems care about.
- 10 Structured State Space Duality The equivalence between selective state space models and a masked form of linear attention, why writing the recurrence as a semiseparable matrix recovers tensor-core throughput, and what the chunked algorithm actually computes.
- 11 The Recurrence-Convolution Duality Why a linear state space layer can be run as a parallel convolution during training and as a constant-memory recurrence at inference, and what the word "linear" is buying.