Group Equivariance and Symmetry

A convolutional layer is not a good architecture because it has fewer parameters. It is a good architecture because it encodes a true fact about images: shifting the input shifts the output, and nothing else changes. That constraint is free information, and a network that has it does not need to spend data discovering it.

Written precisely, a map \(\Phi\) is equivariant to a group \(G\) acting on inputs and outputs if

where \(\rho\) denotes the group action on each space. Invariance is the special case where \(\rho_{\text{out}}\) is trivial, so the output does not change at all. Convolution is exactly the set of linear maps equivariant to translation, a fact that runs in both directions: assume translation equivariance and linearity, and convolution is forced.

Generalising the group

Cohen and Welling made the general construction concrete with G-CNNs, extending convolution from translations to larger discrete groups such as \(p4\) (translations plus 90-degree rotations) and \(p4m\) (adding reflections) (Cohen & Welling, 2016, Group Equivariant Convolutional Networks, arXiv:1602.07576, ICML 2016). A G-convolution correlates the input with a filter transformed by every group element, producing a feature map indexed by the group rather than by position alone.

The accounting is worth being precise about, because it is often stated loosely. A G-CNN does not reduce parameter count; it increases weight sharing, using the same filter across \(|G|\) transformed copies. Capacity per parameter rises, activation memory rises by roughly \(|G|\), and sample efficiency rises because the network no longer has to see a rotated cat to classify a rotated cat.

Continuous groups need more machinery. Steerable CNNs and \(E(3)\)-equivariant networks work with irreducible representations of \(SO(3)\) and combine features using Clebsch-Gordan products, so that rotating the input rotates internal feature vectors correctly by construction. Bronstein et al.'s geometric deep learning programme organises the whole area: CNNs, GNNs, transformers and DeepSets are all "equivariant to some group", differing in which one (Bronstein et al., 2021, arXiv:2104.13478).

Two examples worth holding onto. A graph neural network's message passing is equivariant to node permutation, which is why it can be applied to graphs with no canonical ordering. A transformer without positional encodings is permutation equivariant over tokens, and positional encoding is precisely the deliberate breaking of that symmetry.

Where it pays

The clearest returns are in scientific domains where the symmetry is exact rather than approximate. NequIP, an \(E(3)\)-equivariant network for interatomic potentials, outperformed prior models while using up to three orders of magnitude fewer training configurations (Batzner et al., 2022, E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials, Nature Communications 13:2453). Molecular energy is genuinely invariant to rotation and translation of the molecule; forces are genuinely equivariant. Building that in does not approximate anything, and the data saving is enormous because quantum chemistry labels are expensive.

The pattern generalises: equivariance pays most when labels are costly and the symmetry is exact. It pays least when labels are cheap and the symmetry is approximate.

When it breaks

The dominant counter-argument in vision and language is empirical and hard to dismiss. Vision transformers have no built-in rotation or translation equivariance beyond patch tiling, and they beat carefully constructed equivariant CNNs at scale. Data augmentation buys approximate invariance at zero architectural cost, and approximate invariance is often enough. When data is abundant, learned symmetry beats imposed symmetry, partly because the imposed version is usually wrong at the edges.

It is usually wrong because real symmetries are broken. Images have a preferred vertical: a rotated face is not the same class of thing as an upright one, and enforcing full rotation invariance destroys information the task needs. Text has strict order. Even in chemistry, chirality means full \(O(3)\) invariance is wrong and \(SO(3)\) equivariance plus explicit parity handling is right.

The engineering costs are real too. Equivariant layers are harder to implement, slower per FLOP because their memory access patterns fit accelerators poorly, and much less supported by compilers and kernel libraries than plain matmuls. See graph neural networks and message passing and ML interatomic potentials.