Selectivity and Input-Dependent Parameters

Give an S4 model a simple task: copy a short sequence, ignore a variable number of filler tokens, then reproduce the sequence. It fails. Attention solves this trivially, because a query can look at whichever position holds the content it needs. A linear time-invariant model cannot, and the reason is structural rather than a matter of capacity.

The limitation

In an LTI system, the same convolution kernel is applied at every position regardless of content. The state update \(h_k = \bar{A}h_{k-1} + \bar{B}u_k\) has no mechanism to decide that this particular token is worth remembering and that one is not, because \(\bar{A}\) and \(\bar{B}\) are the same at every step. The model can implement "remember for approximately this long", which is a property of eigenvalues, but not "remember this, forget that", which requires the update to depend on what arrived.

Two canonical tasks isolate this: selective copying, which requires content-based filtering of which tokens to store, and induction heads, which requires retrieving a specific earlier token based on a match. Both are exactly what attention is good at and what an LTI model provably cannot do at fixed state size.

The fix and its cost

Mamba (Gu and Dao, 2024, Mamba: Linear-Time Sequence Modeling with Selective State Spaces, arXiv:2312.00752) makes \(B\), \(C\) and \(\Delta\) functions of the current input, produced by small linear projections. The state matrix \(A\) stays fixed but its effective discretised form \(\bar{A} = \exp(\Delta A)\) becomes input-dependent through \(\Delta\). A large \(\Delta\) for a token means the state resets toward the new input; a small one means the token passes through with little effect. This is a gate, and it recovers exactly the content-dependent forget-and-store behaviour the LTI model lacked.

The cost is immediate and severe: the convolution form is gone. With input-dependent parameters the recurrence is no longer time-invariant, \(\bar{A}^j\) does not factor out, and there is no kernel to FFT. The duality that made these models attractive to train has been spent.

What replaces it is a parallel associative scan. The recurrence \(h_k = \bar{A}_k h_{k-1} + \bar{B}_k u_k\) is an associative operation over pairs, so a work-efficient parallel scan computes all \(L\) states in \(O(\log L)\) sequential depth with \(O(L)\) total work, on a GPU, without ever materialising an \(L\)-length kernel. Training parallelism is recovered by a different route.

When it breaks

Selectivity does not equal attention. The state is still fixed size. A selective model can choose what to store, but it cannot store more than \(N\) dimensions' worth, so tasks needing precise recall of many specific earlier tokens still favour attention. Measured associative-recall accuracy degrades as the number of key-value pairs to remember approaches the state dimension, and this is the practical ceiling that motivates hybrid architectures.

The gate can saturate. Very large \(\Delta\) makes \(\exp(\Delta A)\) collapse toward zero, wiping the state; very small \(\Delta\) makes the token nearly invisible. Both are reachable and both look like the model ignoring parts of the input. Parameterisation of \(\Delta\) through a softplus with a carefully initialised bias is not decoration; it sets where in that range training begins.

Throughput now depends on a custom kernel. The scan is memory-bound, and a naive implementation that materialises the full \((B, L, D, N)\) state tensor in high-bandwidth memory is slower than the attention it replaces. The published speedups assume a fused kernel that keeps state in SRAM and recomputes rather than stores during the backward pass. On hardware or in frameworks where that kernel does not exist, the architecture's advantage largely evaporates.

Long-context claims need the right benchmark. Linear-time scaling in sequence length is a real property, and it does not imply the model uses long context well. Perplexity on long documents improves with context for reasons that include simple local statistics; needle-in-a-haystack and multi-hop retrieval tests are where the fixed-state limitation becomes visible, and they are the tests worth demanding.