Discretising Continuous State Spaces

Train an S4 model on one-second speech clips sampled at 16 kHz, then feed it the same clips at 8 kHz. Most baselines in that comparison either degrade sharply or cannot be run on raw audio at all, because the inputs now arrive twice as far apart. S4 scored 96.3% on the ten-class Speech Commands task at half the frequency with no retraining, by changing one number: its internal step size \(\Delta\) (Gu, Goel and Ré, 2022, Efficiently Modeling Long Sequences with Structured State Spaces, arXiv:2111.00396). That number exists because the layer is defined in continuous time and has to be converted into a recurrence before it can touch a token. The conversion is the subject here.

The Recurrence-Convolution Duality concept takes the discrete matrices \(\bar{A}, \bar{B}\) as given. HiPPO and Selectivity each mention \(\Delta\) in passing. This concept derives where the matrices come from and what the choice of rule does.

Three rules for one integral

The layer is \(h'(t) = A\,h(t) + B\,u(t)\) with state \(h \in \mathbb{R}^N\). Over one step of length \(\Delta\) the exact solution is

The first term is exact. The integral needs an assumption about the input between samples, and each assumption is a discretisation rule.

Zero-order hold assumes \(u\) is constant across the step, which makes the integral closed form:

The bilinear transform (trapezoidal rule) averages the derivative at both ends of the step:

Forward Euler keeps the first-order Taylor term: \(\bar{A} = I + \Delta A\), \(\bar{B} = \Delta B\).

Either way the recurrence is \(h_k = \bar{A}h_{k-1} + \bar{B}u_k\). S4 used bilinear because its state matrix was diagonal plus low rank, where a matrix exponential is expensive and the bilinear inverse is cheap. Once S4D and its successors made \(A\) diagonal, \(e^{\Delta A}\) became elementwise and ZOH cost nothing extra (Gu, Gupta, Goel and Ré, 2022, On the Parameterization and Initialization of Diagonal State Space Models, arXiv:2206.11893).

Where the rules part company

Take a scalar eigenvalue \(a = -1\), a stable continuous system that forgets at unit rate.

At \(\Delta = 0.1\) the three rules nearly agree: ZOH gives \(\bar{a} = e^{-0.1} = 0.9048\), bilinear gives \(0.95/1.05 = 0.9048\) to three places, Euler gives \(0.9\). At \(\Delta = 3\) they diverge completely. ZOH gives \(e^{-3} = 0.050\), almost a full reset. Euler gives \(1 - 3 = -2\), an exploding recurrence. Bilinear gives \((1 - 1.5)/(1 + 1.5) = -0.2\), and as \(\Delta\) grows it tends to \(-1\): at \(\Delta = 100\) it is \(-49/51 \approx -0.96\), a state that flips sign every step and barely decays.

This is the stability picture in miniature. ZOH maps every continuous eigenvalue with negative real part to \(|e^{\Delta\lambda}| < 1\) for any \(\Delta > 0\), and bilinear maps the left half-plane into the unit disc for any \(\Delta\) too, so both are unconditionally stable where Euler requires \(\Delta < 2/|\lambda|\). Stability is not the whole story, though. Only ZOH sends large steps toward forgetting; bilinear sends them toward an oscillating memory.

Step size as a timescale, then as a gate

Under ZOH a real eigenvalue decays by \(e^{\Delta\lambda}\) per token, so the memory half-life in tokens is

S4D initialises \(\Delta\) log-uniformly in \([0.001, 0.1]\), and Mamba draws its initial \(\Delta\) from the same range. With \(|\lambda| = 1\) that spans half-lives from about 693 tokens down to about 7, so a single layer starts with channels tuned to short and long horizons at once. Scaling \(\Delta\) rescales every horizon, which is exactly what the speech experiment exploited.

Mamba makes \(\Delta\) a function of the input through a softplus, and with \(N = 1\), \(A = -1\), \(B = 1\) its Theorem 1 shows the ZOH recurrence becomes \(g_t = \sigma(\mathrm{Linear}(x_t))\), \(h_t = (1-g_t)\,h_{t-1} + g_t\,x_t\), the classic RNN gate (Gu and Dao, 2023, Mamba: Linear-Time Sequence Modeling with Selective State Spaces, arXiv:2312.00752). A large \(\Delta\) resets the state onto the current token and a small one lets the token pass unseen. The derivation leans on ZOH's large-step behaviour, the property bilinear lacks.

When it breaks

The ablations say the rule barely matters. S4D compared bilinear and ZOH directly and found little empirical difference, for example 85.20 against 85.02 on sequential CIFAR. The Mamba paper, by contrast, presents discretisation as the principled foundation of gating. Both can be true: on time-invariant models at moderate \(\Delta\) the rules coincide, while selective models push \(\Delta\) into the regime where they do not.

Implementations mix rules. Mamba's reference selective scan computes \(\bar{A} = \exp(\Delta A)\) but \(\bar{B}u\) as \(\Delta B u\), the Euler form. It is cheaper and works, and it means the continuous-time story describes \(\bar{A}\) more faithfully than \(\bar{B}\).

Resolution invariance needs an underlying signal. Rescaling \(\Delta\) is meaningful for audio or sensor streams sampled from a real clock. Text has no sampling interval, so for language models \(\Delta\) is a learned per-token gate, and claims of resolution invariance do not transfer.

Bad \(\Delta\) ranges look like capacity problems. Initialise too small and channels never forget; too large and they forget every token. Both train, both produce plausible short-range loss, and neither announces itself as a discretisation error.