Long Convolution Models and Implicit Filters

Between S4 and Mamba there was a year in which the most promising attention replacements were not recurrences at all. They were convolutions with kernels as long as the sequence, computed by FFT, with the kernel itself produced by a small neural network rather than stored as weights. The line ran through H3 and Hyena, it produced real language models, and it lost. Knowing why it lost is the fastest way to understand what selectivity is actually for.

A filter you generate instead of store

A convolution with kernel length \(T\) over a sequence of length \(T\) costs \(O(T^2)\) directly and \(O(T \log T)\) through the convolution theorem: transform the input and the kernel, multiply pointwise, transform back. The transform is the easy part. The problem is the kernel, which at \(T = 8192\) would need 8192 parameters per channel, and those parameters would be tied to one sequence length and mostly untrained, since long-range taps see little gradient signal.

Implicit parameterisation removes that. The kernel is the output of a function of position, \(k_t = \gamma(t)\), where \(\gamma\) is a small MLP over a positional encoding, optionally windowed by an exponential decay. Parameter count is now independent of sequence length, the filter is smooth by construction, and the same trained filter can be evaluated at a longer \(T\) at inference. An S4 layer is a special case: its kernel \(\bar{C}\bar{A}^t\bar{B}\) is also a function of \(t\) generated from a handful of parameters, just one derived from a state space rather than from an MLP.

Hyena built its operator by interleaving these implicit long convolutions with multiplicative elementwise gating of projections of the input, recursively, so that the effective mixing depended on the input through the gates while each convolution stayed fixed (Poli et al., 2023, Hyena Hierarchy: Towards Larger Convolutional Language Models, ICML, arXiv:2302.10866). H3 had arrived at a related design from the state space side, stacking a shift SSM and a diagonal SSM with multiplicative interactions specifically to give the layer the shift-and-compare behaviour that induction heads provide in a transformer (Fu et al., 2023, Hungry Hungry Hippos: Towards Language Modeling with State Space Models, arXiv:2212.14052).

Data control through the gate, not the filter

Both designs are linear time-invariant in the convolution and nonlinear in the gates. The distinction matters. Whatever the input says, the filter applies the same taps at the same offsets; the only way content influences mixing is by scaling the result afterwards. Gating can suppress a channel, and it can modulate how two mixed streams combine, but it cannot make the layer look further back for one token than for another.

That is exactly the ceiling Zoology measured. Gated-convolution architectures trailed attention by up to 2.1 perplexity points on the Pile with the deficit concentrated on associative-recall tokens, and they needed model dimension to grow with sequence length to solve multi-query associative recall at all (Arora et al., 2023, arXiv:2312.04927). Selectivity attacked the same problem from inside the operator by making \(\Delta\), \(B\) and \(C\) functions of the token, which lets the layer decide per position whether to absorb or ignore. The cost was the loss of the global convolution and the need for the scan and, later, the chunked SSD kernel.

What survived

The family is not a dead end so much as a set of components that got absorbed. Implicit filter generation persists wherever a length-independent kernel is wanted. The short depthwise causal convolution in front of the Mamba block is a direct descendant of H3's shift component and does real work on local pattern matching. The FFT path still wins for genuinely time-invariant layers at very long lengths, and long-convolution models remain strong in genomics and audio, where the signal really is stationary and the recall demands are mild.

When it breaks

FFT throughput is not what the asymptotics promise. \(O(T\log T)\) with a large constant on hardware optimised for dense matmul frequently loses to \(O(T^2)\) attention at sequence lengths under a few thousand. The crossover is a measured quantity, not a theoretical one, and it moves with every kernel library release.

Autoregressive generation is the awkward case. A convolution is natural over a whole sequence and unnatural one token at a time. Generating from a long-convolution model means either recomputing a growing convolution per token or converting the filter back into a recurrence, and for an implicit MLP filter that conversion is not exact. This is a large part of why the recurrent formulations won for decoder-only language models.

Numerical precision in the frequency domain. FFT convolutions in fp16 accumulate error across the transform, and long kernels with slow decay are the worst case. Implementations that keep the spectral multiply in fp32 pay bandwidth for accuracy, eroding the advantage the FFT was supposed to provide.