Activation Steering and Representation Engineering

Fine-tuning to make a model more cautious costs a training run, a dataset, and a new set of weights to serve. Prompting to the same effect costs tokens on every request and can be argued away by the user. There is a third option that costs a single vector addition per forward pass.

Compute the mean activation difference between two sets of prompts that differ only in the property you care about, then add that difference, scaled, into the residual stream at inference. The behaviour moves. No gradients, no dataset, no retraining.

The construction

Turner et al. introduced activation addition (ActAdd) in its simplest form: contrast the intermediate activations of a prompt pair such as "Love" and "Hate", take the difference as a steering vector, and add it during the forward pass (Steering Language Models With Activation Engineering, arXiv:2308.10248). One pair of data points, no optimisation, and unrelated task performance largely preserved.

For a set of positive prompts \(P^{+}\) and negatives \(P^{-}\), the vector is the difference in means of the layer-\(\ell\) residual stream:

and steering is \(h_\ell \leftarrow h_\ell + \alpha v_\ell\) at every token position, with \(\alpha\) the only knob.

Zou et al. generalised this into representation engineering, a top-down programme that puts population-level representations rather than neurons or circuits at the centre of analysis, and reports control over honesty, harmlessness and power-seeking among others (Representation Engineering: A Top-Down Approach to AI Transparency, arXiv:2310.01405).

The sharpest single result in this line: Arditi et al. found that across 13 open chat models up to 72B parameters, refusal is mediated by a one-dimensional subspace. Erase that direction and the model stops refusing harmful instructions; add it and the model refuses harmless ones (arXiv:2406.11717).

Why this matters beyond control

A steering vector is a falsifiable claim about representation. If adding \(v\) reliably produces the behaviour and ablating it reliably removes it, the concept is linearly represented at that layer, which is a strong statement about geometry that no amount of feature labelling establishes on its own. Steering is therefore an interpretability result, not only an interpretability tool.

The refusal result is also a security finding. A safety behaviour that lives in one direction is a safety behaviour that anyone with weight access can remove with a rank-one edit, which reframes what open-weight release means.

When it breaks

The coefficient has no safe default. Too small does nothing; too large produces the Golden Gate Claude failure mode, where the steered concept intrudes into every output and general capability degrades. The usable band is narrow, model-specific, and layer-specific.

Layer choice matters more than expected. The same vector applied at an early layer versus a middle one produces different effects, and the best layer is found empirically per concept.

Steering vectors generalise unevenly. A vector built from templated contrast pairs often works on similar prompts and fails on paraphrases or other languages, which suggests it has partly captured the template rather than the concept.

Composition is not additive. Two vectors that each work alone frequently interfere when applied together, and there is no established theory that predicts when.