Model Architecture
30 min
Attention Is Memory Retrieval: What the Hopfield View Predicts About Transformers
A 1982 network of 100 binary units could store about 14 memories. In 2020 the same energy idea, with one function changed, turned out to produce transformer attention exactly. The identity is narrow, it is provable, and it predic…