The Kalman Filter and Linear Gaussian State-Space Models

A GPS receiver reports position every second with a few metres of noise. An accelerometer reports motion a hundred times a second, precisely in the short run and drifting badly in the long run. Neither is the truth, and averaging them naively is worse than trusting either at the right moment. The Kalman filter decides, at every step, exactly how much to trust each source, and in a linear Gaussian world its answer is the best possible (Kalman, 1960, A New Approach to Linear Filtering and Prediction Problems, Journal of Basic Engineering 82(1)).

The same machinery sits under a large part of classical time series analysis. ARIMA models, structural trend-and-season models and dynamic regressions can all be written in state-space form, and once written that way, one algorithm filters them, smooths them and evaluates their likelihood (Durbin & Koopman, 2012, Time Series Analysis by State Space Methods, 2^nd ed., OUP).

The model and the two steps

The model has a hidden state \(\alpha_t \in \mathbb{R}^m\) and an observation \(y_t \in \mathbb{R}^p\):

\(Z\) maps state to observation, \(T\) is the transition, \(H\) and \(Q\) are the observation and state noise covariances, and \(R\) maps state noise into the state. Write \(a_{t|t-1}\) and \(P_{t|t-1}\) for the mean and covariance of \(\alpha_t\) given observations up to \(t-1\).

Update. When \(y_t\) arrives, compute the innovation and its covariance, then the gain:

Predict. Push the posterior through the dynamics:

The derivation is one fact about Gaussians. Given the past, \(\alpha_t\) and \(y_t\) are jointly Gaussian with cross-covariance \(P_{t|t-1}Z^\top\), and conditioning a joint Gaussian on one block gives mean \(a_{t|t-1} + P_{t|t-1}Z^\top F_t^{-1}(y_t - Za_{t|t-1})\) and covariance \(P_{t|t-1} - P_{t|t-1}Z^\top F_t^{-1} Z P_{t|t-1}\), which are exactly the update equations. Everything stays Gaussian because every operation is linear.

A worked step, and a bridge to smoothing

Take the local level model: \(m = p = 1\), \(Z = T = R = 1\), observation variance \(H = 4\), level variance \(Q = 1\). The prior is \(a_{t|t-1} = 10\) with \(P_{t|t-1} = 5\), and \(y_t = 13\) arrives. Then \(F_t = 9\), \(K_t = 5/9 \approx 0.56\), the filtered level is \(10 + 0.56 \times 3 = 11.67\), and \(P_{t|t} = 5 \times 4/9 \approx 2.22\). Predicting forward adds \(Q\), so \(P_{t+1|t} \approx 3.22\).

Iterate long enough and \(P\) settles where \(P = 4P/(P+4) + 1\), that is \(P^2 - P - 4 = 0\), giving \(P \approx 2.56\) and a steady-state gain \(K \approx 0.39\). The filtered level then obeys \(a_{t|t} = a_{t-1|t-1} + 0.39\,(y_t - a_{t-1|t-1})\): simple exponential smoothing with \(\alpha = 0.39\), fixed by the signal-to-noise ratio \(Q/H = 0.25\) rather than tuned. This equivalence is the bridge between exponential smoothing and structural models developed in Harvey, 1989, Forecasting, Structural Time Series Models and the Kalman Filter, CUP.

Filtering, smoothing, likelihood

Three different questions share the recursions. Filtering estimates \(\alpha_t\) from data up to \(t\), which is what a live system can know. Smoothing estimates \(\alpha_t\) from the whole sample; the Rauch-Tung-Striebel backward pass computes it as

running from \(t = n-1\) down to 1 (Rauch, Tung & Striebel, 1965, Maximum likelihood estimates of linear dynamic systems, AIAA Journal 3(8)). Using a smoothed state as a model feature is lookahead leakage, which is a common and silent bug in backtests.

Likelihood comes free through the prediction error decomposition. The innovations are independent Gaussians, so

and the unknowns in \(H\), \(Q\) and \(T\) are estimated by maximising it numerically, rerunning the filter at each trial value. Missing observations are handled by skipping the update step, which is one of the most practically useful properties of the whole framework.

When it breaks

Optimality is conditional. Under Gaussian noise the filter gives the exact posterior. Without Gaussianity it is still the best linear estimator in mean squared error, but a nonlinear estimator can do better. Heavy-tailed noise is the usual failure: one outlier gets full weight through \(K_t\) and drags the state for many steps.

Nonlinear dynamics need approximations that are argued over. The extended Kalman filter linearises around the current estimate; the unscented filter propagates sigma points; particle filters sample. Each trades accuracy, cost and robustness differently, and practitioners disagree about the defaults.

Misspecified noise gives confident nonsense. If \(Q\) is set too small the gain collapses, the filter stops listening to data, and \(P_{t|t}\) reports certainty that the errors contradict. Checking that standardised innovations \(F_t^{-1/2}v_t\) look like white noise is the basic diagnostic, and it is often skipped.

Cost and numerics. Each step inverts \(F_t\) and multiplies \(m \times m\) matrices, so cost is \(O(nm^3)\), which bites when the state includes many seasonal dummies. The covariance update \((I - KZ)P\) can lose symmetry and positive definiteness in floating point; the Joseph form or square-root filters fix it at extra cost. Initialising nonstationary states needs diffuse priors, and doing that exactly rather than with a large number is its own literature.