Fourier Analysis for Neural Models

Three unconnected-looking facts have the same explanation. Long-convolution sequence models are subquadratic. Coordinate MLPs produce blurry outputs unless their inputs are preprocessed. Extending a model's context window can be done by rescaling numbers inside its position encoding. All three are Fourier-domain statements, and seeing that turns three tricks into one idea.

The convolution theorem, and where the speed comes from

For sequences of length \(n\),

Convolution in time is pointwise multiplication in frequency. Direct convolution with a kernel as long as the sequence is \(O(n^2)\); FFT, multiply, inverse FFT is \(O(n \log n)\). This is the mechanism behind the whole family of long-convolution sequence models: S4 reaches state of the art across Long Range Arena including Path-X at length 16k, closing much of the gap to transformers while generating 60× faster (Gu, Goel and Ré, ICLR 2022, arXiv:2111.00396). The same theorem is what makes Fourier neural operators cheap: parameterise the kernel directly in frequency space, keep the lowest modes, and every layer sees the entire domain rather than a local receptive field.

Two practical caveats travel with it. The FFT computes circular convolution, so causal use requires zero-padding to \(2n\) and discarding the wrapped half. And the FFT needs the whole sequence at once, which is why these models carry a separate recurrent formulation for autoregressive decoding.

Spectral bias

A standard MLP fitting a function of low-dimensional coordinates learns low frequencies first and high frequencies extremely slowly, sometimes never. This is not a training bug; it follows from the neural tangent kernel of such a network, whose eigenvalues decay rapidly with frequency, so high-frequency components have vanishing effective learning rates.

The fix is to change the kernel rather than the network. Mapping the input through random Fourier features \(\gamma(v) = [\cos(2\pi B v), \sin(2\pi B v)]\) turns the effective NTK into a stationary kernel with a bandwidth set by the scale of \(B\), which can then be matched to the frequency content of the target function (Tancik et al., NeurIPS 2020, arXiv:2006.10739). This is why every neural radiance field and implicit surface model applies a positional encoding to coordinates before the first linear layer: without it, the network is a low-pass filter.

Position encoding as a frequency bank

Sinusoidal position encodings and rotary embeddings are both banks of oscillators at geometrically spaced frequencies. RoPE encodes absolute position by rotating each two-dimensional slice of a query or key by an angle proportional to position, with rotation rates \(\theta_i = 10000^{-2i/d}\), so that the attention dot product depends only on the relative offset (Su et al., 2021, arXiv:2104.09864). Low-index dimensions rotate fast and encode fine local distinctions; high-index dimensions rotate slowly with wavelengths longer than the training context.

Context extension methods are then straightforward frequency edits. Position interpolation compresses all wavelengths so that a longer sequence fits into the rotation range the model saw in training. NTK-aware scaling instead changes the base of the geometric progression, stretching low-frequency dimensions more than high-frequency ones, on the argument that fast-rotating dimensions have already seen every phase they will ever see and only the slow ones are extrapolating. Both are decisions about which frequency bands to distort.

Where the Fourier view stops helping

Frequency is only meaningful on a regular grid. FFT-based methods want uniformly sampled sequences. Irregular time series, event streams and variable-rate sensor data need resampling first, and the resampling error is often larger than whatever the spectral method saves.

Global kernels have no locality prior. A convolution parameterised in frequency space is global by construction, which is powerful and also means it cannot cheaply express "attend only to the previous token" without spending many modes on it.

Numerical precision. FFTs accumulate error across \(\log n\) butterfly stages and are unfriendly to fp16; long-convolution implementations typically run the transform in fp32 even inside a mixed-precision model.

Content-dependence is missing. The decisive advantage of attention is that its mixing weights depend on the input. A fixed spectral kernel is the same for every sequence, which is why these architectures usually appear in hybrids with a few attention layers rather than as replacements.