Hopfield Networks and Attractor Memory

A network of 100 binary units, programmed in a single pass of Hebbian outer products with no gradient descent anywhere, will return a stored pattern correctly from a corrupted fragment as long as you store no more than about fourteen of them. Store twenty and it does not degrade gracefully; it returns blends and spin-glass states that were never stored, and the earlier memories go with them. The retrieval is the famous part. The cliff is the part that shaped forty years of follow-up work, and it is the reason the modern variants exist.

The energy that makes a memory a fixed point

Store \(P\) patterns \(\xi^1,\dots,\xi^P \in \{-1,+1\}^N\) in a symmetric weight matrix by Hebbian outer products, with the diagonal zeroed:

and define an energy over network states \(s \in \{-1,+1\}^N\):

Update one unit at a time, \(s_i \leftarrow \operatorname{sign}\!\left(\sum_j w_{ij}s_j\right)\). Because the weights are symmetric and the diagonal is zero, every flip that the rule accepts lowers \(E\) or leaves it unchanged, so \(E\) is a Lyapunov function and the dynamics cannot cycle: they fall into a local minimum and stop (Hopfield, 1982, Neural networks and physical systems with emergent collective computational abilities, PNAS 79(8)).

That is the whole trick. If each stored pattern sits at a local minimum, then the set of states that flow into it is its basin of attraction, and running the dynamics from a corrupted version of \(\xi^\mu\) is recall. Hopfield's description of the behaviour is still the clearest one available: the network "correctly yields an entire memory from any subpart of sufficient size". Memory becomes content-addressable, with no index and no address decoder, and the work in 2024 that this line of research won the Nobel Prize in Physics for was exactly this reframing of memory as the settling of a physical system (Royal Swedish Academy of Sciences, 2024, The Nobel Prize in Physics 2024).

Why the capacity is linear in \(N\), and small

Put the network in state \(\xi^1\) and look at the local field on unit \(i\):

The first term is the signal, which always agrees with the bit you want. The second is crosstalk from the other \(P-1\) patterns, a sum of roughly \(N(P-1)\) random signs scaled by \(1/N\), so its standard deviation is about \(\sqrt{(P-1)/N}\). The bit is stable while the signal beats the noise, which is why capacity scales with \(N\) and why the ratio \(\alpha = P/N\) is the control parameter rather than \(P\) itself.

Hopfield's own Monte Carlo runs at \(N = 30\) and \(N = 100\) put the usable load at roughly \(0.15N\) before recall degraded badly. The statistical mechanics came later: analysing the model as a spin glass with the replica method gives a first-order transition at a critical load of \(\alpha_c \approx 0.138\), below which retrieval states exist and above which they vanish altogether (Amit, Gutfreund and Sompolinsky, 1985, Storing infinite numbers of patterns in a spin-glass model of neural networks, PRL 55(14); the letter itself quotes the threshold only as \(\alpha_c \gtrsim 0.14\), and 0.138 is the replica-symmetric value usually cited afterwards).

Demanding more than approximate recall costs a logarithm. If every stored memory must be an exact fixed point with probability approaching one, the number of patterns can be at most about \(N/(4\log N)\), and about \(N/(2\log N)\) if you only need most of them to be exact (McEliece, Posner, Rodemich and Venkatesh, 1987, The capacity of the Hopfield associative memory, IEEE Trans. Inf. Theory 33(4)). At \(N = 4096\) those two criteria differ by a factor of four and a half: 565 patterns by the \(0.138N\) rule, 123 by the exact-recall rule.

Measured as information, this is poor. Storing \(0.138N\) patterns of \(N\) bits each in \(N^2\) weights is about 0.14 bits per weight, and the weights are real-valued.

What the network returns when you ask for too much

The stored patterns are never the only minima. Three other families exist and all of them are reachable.

Reversed patterns. \(E(-s) = E(s)\), so \(-\xi^\mu\) is a minimum whenever \(\xi^\mu\) is. A query closer to the inverse than to the pattern retrieves the inverse.

Mixture states. Odd combinations such as \(\operatorname{sign}(\xi^1 + \xi^2 + \xi^3)\) are typically stable. They are plausible-looking states that were never stored, which is the failure mode worth remembering: the network does not report "not found", it reports a confident blend.

Spin-glass states. Above \(\alpha_c\) the landscape is dominated by minima uncorrelated with any stored pattern. Retrieval does not get noisier as you cross the threshold, it stops.

When it breaks

Correlated patterns merge their basins. The crosstalk calculation assumes random signs. Real data is not random: store ten near-duplicate images and their individual minima fuse into one, so the network recalls the average and the distinctions are gone. Every published capacity figure is for random patterns and is an upper bound on what structured data will give you.

Asymmetric weights remove the guarantee. Symmetry is what makes \(E\) decrease monotonically. Learn \(w_{ij} \neq w_{ji}\) and the dynamics can cycle indefinitely; there is no energy to descend and no convergence theorem. This matters when reading the modern Hopfield literature, because a transformer's separate query and key projections break exactly this condition.

The cliff is first-order. Capacity does not degrade smoothly with load. Crossing \(\alpha_c\) loses every memory at once, which makes the model unusable as an incrementally-filled store unless you know \(P\) in advance.

Capacity, not compression. A Hopfield network does not store patterns more cheaply than a list would; \(N^2\) weights hold \(0.138N\) patterns. What it buys is addressing by content, and that is the only thing it buys.