The Residual Stream: The Transformer's Shared Memory Bus

In August 2020, a blogger writing as nostalgebraist tried something with GPT-2 that had no obvious reason to work. The model's final operation multiplies its last hidden state by an unembedding matrix to produce vocabulary logits; nostalgebraist applied that same matrix to the hidden states halfway through the network, and to every other layer while he was at it. The result should have been noise, because nothing in the training objective asks intermediate layers to be legible in vocabulary space. Instead, out came a coherent trajectory: rough guesses in early layers, sharpening steadily, with the model's final answer often visible and highly ranked twenty layers before the end (nostalgebraist, 2020, interpreting GPT: the logit lens).

That experiment only makes sense under a particular reading of the architecture, one where the thing flowing through the network is not "layer 24's output" but a single persistent vector that every layer incrementally edits, in a shared coordinate system that the embedding sets up and the unembedding reads out. Anthropic's interpretability team gave that reading its name and its algebra a year later: the residual stream, a communication channel of width \(d_{\text{model}}\) that attention heads and MLPs read from and write to, with the layers as peripherals on a bus rather than stages in a pipeline (Elhage et al., 2021, A Mathematical Framework for Transformer Circuits).

Why this matters: The residual stream is the frame in which almost all mechanistic knowledge about transformers is expressed. The logit lens, activation patching, steering vectors, sparse autoencoder features, superposition, model editing: every one of these techniques is an operation on the stream, and none of them is even stateable in the layers-as-pipeline picture. If you want to reason about what a language model is doing internally, this is the coordinate system the field settled on.

TL;DR

• A transformer's forward pass at each position is \(x_{l+1} = x_l + F_l(x_l)\): pure addition. The final state is literally the embedding plus the sum of every sublayer's output, so any layer's contribution to the logits can be isolated, measured, and ablated (Elhage et al., 2021).
• The stream is the bottleneck by design: in GPT-2 small it is 768 dimensions wide, shared by 12 attention heads per layer that each read and write through low-rank (64-dimensional) channels, which is why head outputs are best understood as edits to subspaces, not replacements of state.
• Depth behaves like iterative refinement, not feature hierarchy: predictions decoded from intermediate layers improve roughly monotonically, a phenomenon measurable with an affine probe per layer across models up to 20B parameters (Belrose et al., 2023, arXiv:2303.08112); the idea that residual networks refine rather than transform dates to ResNet analysis (Jastrzębski et al., 2018, arXiv:1710.04773).
• The stream carries far more concepts than it has dimensions, by storing features as nearly-orthogonal directions in superposition; sparse autoencoders decompose it, at the cost of training dictionaries with up to 34 million features for one middle layer of one production model (Templeton et al., 2024).
• Residual stream norms grow roughly exponentially with depth, around 4.5% per layer in GPT2-XL, so late layers must "shout" to move the state and early-layer writes fade in relative terms (Heimersheim and Turner, 2023).
• None of this was designed. Residual connections entered the architecture as a trainability fix for vanishing gradients; the memory-bus computational structure is an emergent consequence the field spent years reverse-engineering.

At a Glance

One block of a decoder-only transformer, drawn the way the interpretability literature draws it: the stream runs vertically untouched, and each sublayer branches off, computes, and merges its result back by addition.

flowchart TB
  X0["Residual stream x_l"] --> LN1["LayerNorm"]
  LN1 --> ATT["Attention heads: read other positions"]
  ATT --> ADD1["Add write back to stream"]
  X0 --> ADD1
  ADD1 --> LN2["LayerNorm"]
  LN2 --> MLP["MLP: read and transform this position"]
  MLP --> ADD2["Add write back to stream"]
  ADD1 --> ADD2
  ADD2 --> XL["Residual stream x_l+1"]
  classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  class X0,XL,ADD1,ADD2 teal
  class LN1,LN2 slate
  class ATT,MLP purple

[IMAGE: A hardware-style schematic of a motherboard bus: a wide horizontal trace labelled "residual stream, d_model wires" with small peripheral cards plugged in along its length labelled "head 0.3", "MLP 5", "head 9.1", each card showing a thin "read" tap and a thin "write" tap into the bus. Caption: "Layers are peripherals; the stream is the bus they communicate through."]

Before the Stream: A Trainability Hack Becomes an Architecture

Nothing about the residual stream was planned as a computational substrate. Its ancestry is a series of fixes for the same practical problem: deep networks would not train. LSTM's cell state (1997) was the first widely used "information highway" through time, a path along which gradients could flow unmodified. Highway networks made the idea depth-wise with learned gates (Srivastava et al., 2015, arXiv:1505.00387), and ResNet stripped the gates away entirely: just add the block's input to its output, \(x + F(x)\), and suddenly 152-layer networks trained where 30-layer ones had choked (He et al., 2015, arXiv:1512.03385).

Then came the reinterpretations, and they arrived before the transformer did. Veit et al. showed that a ResNet behaves like an ensemble of exponentially many paths of different lengths, and that almost all of the training gradient flows through surprisingly shallow paths, around 10 to 34 layers in a 110-layer network; deleting single layers barely hurt, which no pipeline story can explain (Veit et al., 2016, arXiv:1605.06431). Jastrzębski et al. made the refinement story quantitative: residual blocks mostly nudge features along the loss gradient, iteratively improving a shared representation rather than building a new one per layer (Jastrzębski et al., 2018, arXiv:1710.04773).

The 2017 transformer inherited residual connections as standard practice (Vaswani et al., 2017, arXiv:1706.03762), and one further change completed the modern picture. The original design applied LayerNorm after each addition, placing a normalisation inside the skip path itself; moving it to the front of each sublayer (pre-LN) left the identity path genuinely untouched from embedding to unembedding, and was adopted across GPT-2-era models for its training stability (Xiong et al., 2020, arXiv:2002.04745). It is exactly this uninterrupted additive path that makes the logit lens and everything after it possible; on post-LN models, the same tricks work far less cleanly.

timeline
    title From gradient fix to computational substrate
    1997 : LSTM cell state gives gradients an unmodified path through time
    2015 : Highway networks gate the depth-wise shortcut; ResNet removes the gates
    2016 : Veit et al. show ResNets act as ensembles of shallow paths
    2017 : The transformer ships with residual connections as default plumbing
    2020 : Pre-LN standardises the clean identity path; the logit lens reads it
    2021 : Elhage et al. name the residual stream and give it an algebra
    2023 : Tuned lens makes layerwise decoding rigorous; SAEs begin decomposing the stream
    2024 : Dictionary learning scales to 34M features on a production model

A Bus, Not a Pipeline

The algebra: everything is a sum

Fix one token position and follow its vector. With pre-LN blocks, the forward pass is

and unrolling the recursion gives the statement that carries the whole framework:

the embedding plus every attention write \(a_l\) plus every MLP write \(m_l\), all living in the same \(d_{\text{model}}\)-dimensional space. The logits are a linear read of this sum (LayerNorm's scaling aside), so each write's direct contribution to any token's logit is just a dot product with that token's unembedding row. This is why "which component pushed the model toward Paris" is a well-posed, computable question rather than a metaphor, and it is the calculation behind direct logit attribution, one of the workhorse techniques of the field (Elhage et al., 2021).

Two things temper the linearity before it becomes too good to be true. LayerNorm rescales by the running norm, a mild nonlinearity that lens methods must calibrate away. And the decomposition is exact only for direct paths; a write also matters through every downstream layer that reads it, which is where the real circuit analysis lives.

Reads and writes: bandwidth, addresses, subspaces

The bus metaphor earns its keep in the details. An attention head does not see the stream whole: its query, key, and value projections are rank-64 reads (in GPT-2 small; \(d_{\text{head}} = d_{\text{model}} / n_{\text{heads}}\) generally), and its output projection is a rank-64 write. A head is thus a narrow-bandwidth device that moves information between positions: what gets read is chosen by the QK circuit, what gets written by the OV circuit, and the two are separately analysable low-rank matrices. The MLP, by contrast, is a wide same-position device: it reads the full stream through \(W_{\text{in}}\) (768 to 3072 dimensions), applies its nonlinearity, and writes back through \(W_{\text{out}}\). Roughly two thirds of a GPT-class model's parameters sit in these MLP read/write matrices.

Because every device writes into the same 768 dimensions, the stream functions as shared memory with no memory protection: components communicate by convention, using directions as addresses. One head writes a "subject is a landmark" direction; a later MLP's input weights have learned to fire on that direction; nothing but training pressure keeps their addresses aligned. [IMAGE: Cross-section of the stream drawn as a 768-cell strip, with three overlaid translucent arrows at different angles labelled as feature directions ("landmark", "past tense", "Python code"), deliberately non-axis-aligned and slightly overlapping. Caption: "Features are directions, not dimensions: the basis is arbitrary, and near-orthogonal directions let the stream carry more features than it has wires."]

Composition of this kind, one component's write serving as another's read, is the definition of a circuit, and the cleanest documented example is the induction head pair: a previous-token head writes "token B followed A" into B's stream position, and an induction head at a later layer reads that annotation to continue the pattern "A B ... A" with B, a mechanism identified as a major carrier of in-context learning (Olsson et al., 2022, In-context Learning and Induction Heads).

Iterative refinement, measured

The logit lens made refinement visible but was brittle: on several model families the raw unembedding produces garbage at early layers because each model's stream drifts through its own changing basis. The tuned lens repairs this with a small affine probe per layer, trained to translate that layer's state into final-layer coordinates, and the repaired trajectories are consistent across families up to 20B parameters: prediction quality improves layer by layer, smoothly, with most of the final answer often in place well before the end (Belrose et al., 2023, arXiv:2303.08112). The transformer, on this evidence, spends its depth polishing one evolving guess, exactly as the ResNet analyses predicted.

Refinement has a loudness problem, though. The norm of the stream grows roughly exponentially with depth, about 4.5% per layer in GPT2-XL; a plausible mechanism is that LayerNorm makes existing features hard to cancel but easy to overshadow, so each layer writes slightly louder than the accumulated past (Heimersheim and Turner, 2023). Any analysis that compares writes across depth has to normalise for this, and steering interventions that work at layer 6 can be inaudible at layer 30.

[IMAGE: Semi-log line chart of residual stream L2 norm (y-axis, log scale) against layer index 0 to 47 for GPT2-XL, forming a near-straight line, with a dashed reference line at 4.5% per-layer growth and an annotation arrow marking where a fixed-magnitude steering vector added at layer 6 falls below 1% of the stream norm. Caption: "Writing louder than the past: exponential norm growth means depth is also a loudness gradient."]

Superposition: more features than wires

A 768-wide bus carrying a model's entire working state should feel impossibly narrow, and it is, if features get one dimension each. They do not. Because real features are sparse (few are active on any given token), the stream can store vastly more features than dimensions as nearly-orthogonal directions, tolerating small interference in exchange for capacity; Anthropic's toy-model work showed networks doing precisely this under sparsity pressure (Elhage et al., 2022, Toy Models of Superposition, arXiv:2209.10652). The practical consequence is that individual stream dimensions (and individual neurons) are polysemantic and largely meaningless alone; the stream has no privileged basis, since nothing in the architecture distinguishes coordinate axes from any rotation of them. Recovering the real variables requires dictionary learning: sparse autoencoders trained to re-express the stream as sparse combinations of learned directions, which at production scale meant dictionaries of 1M, 4M, and 34M features on Claude 3 Sonnet's middle layer, yielding features for everything from famous landmarks to code bugs, and enabling the steering stunt of Golden Gate Claude (Templeton et al., 2024, Scaling Monosemanticity).

Seeing It in Motion

The induction circuit, played out over the stream at two positions. Nothing is "passed to the next layer"; everything is written once and read later by whichever component learned to look.

sequenceDiagram
  participant SA as Stream at position A
  participant SB as Stream at position B
  participant PTH as Previous-token head (early layer)
  participant IND as Induction head (later layer)
  Note over SA,SB: Prompt so far: ... A B ... A
  PTH->>SA: read token identity at A
  PTH->>SB: write "preceded by A" into B's stream
  Note over SB: Annotation sits in the stream, untouched, across layers
  IND->>SA: query from second A: "who was preceded by A?"
  IND->>SB: key matches the stored annotation
  IND->>SA: write "predict B next" into A's stream
  Note over SA: Unembedding reads the final sum; B tops the logits

And the lifecycle of a single prediction under the tuned lens, decoded at successive depths:

stateDiagram-v2
  [*] --> Embedded: token plus position written
  Embedded --> Contextualised: early heads merge local context
  Contextualised --> Retrieved: mid layers write recalled associations
  Retrieved --> Refined: late MLPs sharpen and suppress alternatives
  Refined --> Decoded: unembedding reads the accumulated sum
  Decoded --> [*]

[IMAGE: A tuned-lens heatmap: layers 0 to 47 on the y-axis, next-token candidates on the x-axis, cell colour showing probability, with the correct token's column visibly warming from noise at layer 5 to dominant from roughly layer 30 onward. Caption: "Refinement made visible: the answer assembles gradually in the stream, and the last third of the network mostly consolidates."]

By the Numbers

Sources: GPT-2 dimensions from the released model configurations (Radford et al., 2019); norm growth from Heimersheim and Turner, 2023; path statistics from Veit et al., 2016; lens scale from Belrose et al., 2023; SAE sizes from Templeton et al., 2024.

A Concrete Example

A four-dimensional toy stream makes the read/write arithmetic replayable on paper. Suppose the unembedding rows for two candidate tokens are \(u_{\text{Paris}} = (0, 0, 1, 0)\) and \(u_{\text{London}} = (0, 0, 0, 1)\), and the prompt is "The Eiffel Tower is in". Track the stream at the final position:

1. Embedding writes the token "in": \(x = (1, 0, 0, 0)\). Direct logits: Paris \(0\), London \(0\). Nothing known yet.
2. An attention head reads from the "Eiffel" position (its QK circuit matched "landmark mentioned earlier") and writes a landmark-identity direction through its OV circuit: adds \((0, 2, 0, 0)\). Stream: \((1, 2, 0, 0)\). Direct logits still \(0\) each: this write is an address, not an answer; it points at nothing in vocabulary space.
3. An MLP reads the stream, and one learned direction in \(W_{\text{in}}\) fires on the pattern "landmark = Eiffel, relation = located-in". Its write through \(W_{\text{out}}\) adds \((0, 0, 3, 0.5)\): strongly Paris, faintly London (France and England co-occur in location contexts; interference is the price of superposition). Stream: \((1, 2, 3, 0.5)\).
4. Unembedding reads the sum. Paris logit: \(x \cdot u_{\text{Paris}} = 3\). London: \(0.5\). Softmax over just these two: \(e^3 / (e^3 + e^{0.5}) \approx 0.924\). The model says Paris.
5. Now patch. Zero the attention write from step 2 and replay steps 3 and 4: the MLP's trigger direction no longer fires, its write never lands, and the final stream is \((1, 0, 0, 0)\) with logits \(0\) and \(0\): the prediction collapses to chance. That counterfactual difference, 0.924 versus 0.5, is activation patching's measurement, and scaling this exact procedure up is how causal tracing located factual recall in mid-layer MLPs of real models (Meng et al., 2022, Locating and Editing Factual Associations in GPT, arXiv:2202.05262).

The two-step structure, attention fetching an address and an MLP dereferencing it into content, is not an artefact of the toy; it is the documented anatomy of factual recall circuits.

[IMAGE: Five-row table graphic mirroring the worked example: each row shows the 4-dimensional stream vector after a step, rendered as four coloured cells with values, alongside the two candidate logits as small bars; the final row repeats step 4 with the attention write zeroed and the Paris bar collapsed. Caption: "One patch, one collapsed logit: the counterfactual difference is the measurement."]

Where It Breaks

No privileged basis, so no honest neuron stories

Every claim of the form "dimension 217 encodes negation" is suspect on architectural grounds: rotate the stream and its interacting weights and the model computes identically while dimension 217 encodes nothing. Directions are meaningful; coordinates are not. Add superposition and even direction-level stories degrade: features overlap, and a probe that reads one cleanly on-distribution reads its interfering neighbours off-distribution. This is the standing epistemic hazard of the whole enterprise, and the reason dictionary learning exists.

[IMAGE: Histogram of path lengths through a 110-layer residual network (binomial-shaped, centred near 55) overlaid with a second distribution showing where gradient magnitude actually concentrates (10 to 34 layers), the two distributions visibly disjoint in their mass. Caption: "Most paths exist; short paths train the network. Veit et al.'s lesion results follow directly."]

The stream fights back when you cut it

Ablation, the field's favourite causal tool, assumes removing a component removes its function. Transformers violate the assumption: knock out an attention layer and downstream layers compensate, restoring much of the lost effect, a phenomenon documented as the Hydra effect, alongside late MLPs whose job is partly to downregulate the leading prediction (McGrath et al., 2023, The Hydra Effect, arXiv:2307.15771). Self-repair means a small measured ablation effect does not prove a small causal role, and every clean circuit diagram is drawn over an organism that reroutes around damage.

Lenses can see things the model does not use

A probe finding "the answer is present at layer 20" does not establish that the model's own layer-30 computation reads it from where the probe did. The tuned lens's causal checks address this partially, but the general trap remains: decoding methods measure information availability, not information use, and the gap between them has repeatedly embarrassed interpretability claims. The original logit lens's family-specific brittleness is the same lesson from the other side; a tool that happens to work on GPT-2 is a fact about GPT-2 until proven otherwise (Belrose et al., 2023).

Norm growth and stream crowding

Exponential norm growth means interventions do not transfer across depth: a steering vector calibrated at one layer is drowned out later, and comparative statements about "how much" a layer contributes require careful normalisation (Heimersheim and Turner, 2023). There is also a capacity wall: the stream width caps how many strongly-active features coexist before interference costs bite, one reason scaling laws push \(d_{\text{model}}\) up with everything else and a candidate mechanism in several long-context failure stories.

The clean algebra is pre-LN algebra

Post-LN models, and architectures that renormalise or gate the skip path, break the "final state equals sum of writes" identity; direct logit attribution and lens methods get muddier. The frame is powerful partly because the dominant architecture family happens to keep the identity path clean; it is a fact about a design lineage, not a theorem about deep learning.

Alternative Designs

The comparison's punchline: pure addition won not because interference is harmless but because it is cheap, and models learn to manage the shared bus well enough that the alternatives' bookkeeping never pays for itself at scale.

How It Is Used in Practice

The stream frame is the daily working environment of mechanistic interpretability. Tooling like TransformerLens exposes models as hook points on the stream precisely so that patching, ablation, and direct logit attribution are one-liners. Causal tracing over stream states located factual associations in mid-layer MLPs and turned that into ROME's targeted weight edits (Meng et al., 2022). Steering vectors operationalise write-access for humans: compute the stream-difference between contrasting prompts and add it during inference to shift sentiment or topic without touching a weight (Turner et al., 2023, Activation Addition, arXiv:2308.10248); SAE features refine the same intervention to single-concept precision, Golden Gate Claude being the public demonstration (Templeton et al., 2024). Anthropic's attribution-graph work pushes toward tracing full computational paths through stream features on frontier models (Lindsey et al., 2025, On the Biology of a Large Language Model), and stream probes are increasingly deployed as monitors, reading safety-relevant features mid-inference. On the engineering side, the refinement picture licenses early exit (decode when the intermediate state has converged) and motivated the logit-lens-style self-drafting used in some speculative decoding research; the frame pays rent outside interpretability proper (see the residual stream and transformer anatomy track for the concept-level components).

Insights Worth Remembering

1. The stream is the object; layers are edits. The final representation is the embedding plus a sum of a few dozen writes. Any question about model internals that cannot be phrased as "who wrote what, and who read it" is probably not yet a well-posed question.
2. Attention moves information between positions; MLPs transform it in place. Heads are narrow cross-position taps (rank 64 in GPT-2 small); MLPs are wide same-position processors holding most of the parameters. Circuits are compositions of the two through the shared bus.
3. Depth is refinement, not hierarchy. Lens trajectories and ResNet path analyses agree: the network polishes one evolving state, and much of the answer exists early. "Layer 20 features feed layer 21" is the wrong default picture; "layer 21 edits a document layer 20 also edited" is the right one.
4. A trainability hack became the computational substrate. Residual connections were adopted to fight vanishing gradients; the memory-bus semantics were discovered, not designed, five years later. Architecture choices have consequences their authors never intended, and interpretability inherits whichever ones training found useful.
5. Superposition is the price of a narrow bus. More features than dimensions means interference, polysemantic neurons, and no privileged basis; every clean single-neuron story should be presumed false until a direction-level or dictionary-level analysis backs it.
6. The stream is shared memory without memory protection. Nothing enforces the conventions components use to address one another; training pressure aligns them, and interventions (steering, editing, patching) work by exploiting exactly that unenforced openness.
7. Measured effects understate causal structure. Self-repair means ablations lie small; availability-versus-use means probes lie large. The stream frame makes causal questions statable; it does not make them easy.

Open Questions

• How much stream capacity do frontier models actually use? Superposition theory predicts a sparsity-dependent capacity; measured feature counts (34M for one layer of one mid-size model) are dictionary sizes, not proofs of what the model uses. Whether stream width is a binding constraint at frontier scale, and whether it explains specific failure modes, is unresolved.
• Do SAE features carve the stream at its joints? Dictionary learning finds a sparse basis, demonstrably useful for steering; whether these are the model's own computational variables or a useful re-description is actively contested, with attribution-graph work the current best attempt to close the gap (Lindsey et al., 2025).
• Why does norm growth settle near 4.5% per layer? The overshadow-rather-than-cancel mechanism is a hypothesis with supporting evidence in GPT-2-family models; whether the rate is universal, what sets it, and whether it is functional or incidental are open (Heimersheim and Turner, 2023).
• Does the frame survive the architecture's evolution? MoE routing, state-space hybrids, and aggressive KV compression all perturb the clean additive picture in different ways. The stream algebra is a fact about one architectural lineage; how much of the accumulated interpretability toolkit transfers is an empirical question being answered piecemeal.
• Can stream monitoring become an engineering discipline? Probes reading deception- or jailbreak-relevant directions during inference exist in research form. Whether they can be made robust to distribution shift and adversarial pressure, at acceptable false-positive rates, is the open question separating interpretability demos from deployed safety infrastructure.

Sources and Further Reading

1. Elhage, N., Nanda, N., Olsson, C., et al. (2021). "A Mathematical Framework for Transformer Circuits." Anthropic. transformer-circuits.pub
2. nostalgebraist (2020). "interpreting GPT: the logit lens." LessWrong. lesswrong.com
3. Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., & Steinhardt, J. (2023). "Eliciting Latent Predictions from Transformers with the Tuned Lens." arXiv:2303.08112
4. He, K., Zhang, X., Ren, S., & Sun, J. (2015). "Deep Residual Learning for Image Recognition." arXiv:1512.03385
5. Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015). "Highway Networks." arXiv:1505.00387
6. Veit, A., Wilber, M., & Belongie, S. (2016). "Residual Networks Behave Like Ensembles of Relatively Shallow Networks." NeurIPS 2016. arXiv:1605.06431
7. Jastrzębski, S., Arpit, D., Ballas, N., Verma, V., Che, T., & Bengio, Y. (2018). "Residual Connections Encourage Iterative Inference." ICLR 2018. arXiv:1710.04773
8. Xiong, R., et al. (2020). "On Layer Normalization in the Transformer Architecture." ICML 2020. arXiv:2002.04745
9. Olsson, C., Elhage, N., Nanda, N., et al. (2022). "In-context Learning and Induction Heads." Anthropic. transformer-circuits.pub
10. Elhage, N., Hume, T., Olsson, C., et al. (2022). "Toy Models of Superposition." Anthropic. arXiv:2209.10652
11. Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022). "Locating and Editing Factual Associations in GPT." NeurIPS 2022. arXiv:2202.05262
12. McGrath, T., et al. (2023). "The Hydra Effect: Emergent Self-repair in Language Model Computations." arXiv:2307.15771
13. Heimersheim, S., & Turner, A. (2023). "Residual stream norms grow exponentially over the forward pass." AI Alignment Forum. alignmentforum.org
14. Turner, A. M., Thiergart, L., Leech, G., Udell, D., Mini, U., & MacDiarmid, M. (2023). "Activation Addition: Steering Language Models Without Optimization." arXiv:2308.10248
15. Templeton, A., et al. (2024). "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet." Anthropic. transformer-circuits.pub
16. Lindsey, J., et al. (2025). "On the Biology of a Large Language Model." Anthropic. transformer-circuits.pub
17. Vaswani, A., et al. (2017). "Attention Is All You Need." NeurIPS 2017. arXiv:1706.03762

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.