Model Architecture
23 min
The Residual Stream: The Transformer's Shared Memory Bus
Stop reading the transformer as a pipeline of 96 layers, each transforming the output of the last. Read it as one shared communication channel that every attention head and MLP merely edits. This one reframing, formalised by inte…