Contrastive Divergence and the Negative Phase
How energy-based training approximates an intractable expectation with a short MCMC chain, what bias that introduces, and why the same positive-negative structure appears in contrastive learning and reward modelling.
The gradient of an energy-based model's log-likelihood is a difference of two expectations: one over the data, which is a minibatch average, and one over the model, which requires sampling from a distribution you can only evaluate up to a constant. Contrastive divergence is the approximation that made these models trainable in the first place, and the pattern it introduced outlived the models themselves.
The approximation
Rather than running an MCMC chain to equilibrium from a random start, initialise the chain at a data point and run \(k\) steps, often \(k=1\). Use the resulting sample as the negative example. Hinton's original argument was that starting from the data puts the chain in a high-probability region already, so a short run produces samples that are informative about the local shape of the model distribution even though they are far from equilibrium.
The update becomes: lower the energy of the data point, raise the energy of the point the chain drifted to. If the model already matches the data, the chain does not drift systematically and the two terms cancel. If the model puts mass where the data is not, the chain moves there and gets penalised. It is a local correction rather than a global one, and that is both why it works and why it is biased.
What the bias is
CD does not follow the gradient of the log-likelihood. It follows the gradient of a different objective, and the two share a fixed point only under conditions that do not hold in general. The practical consequences are specific: the model learns the shape of the distribution near the data well and can leave far regions arbitrarily wrong, since no chain ever visits them to push their energy up.
Persistent contrastive divergence addresses this by keeping a pool of chains that continue across parameter updates rather than restarting from data. The chains have many more steps of history, so they reach regions the model wrongly favours, and they track the model as it changes. The cost is a new failure mode: if the parameters move faster than the chains mix, the chains lag behind the current model and provide negatives for a distribution that no longer exists.
The same shape elsewhere
Strip the MCMC and the structure is recognisable across modern machine learning. Noise-contrastive estimation replaces model samples with samples from a fixed known noise distribution and trains a classifier to separate them, which turns density estimation into supervised learning. Contrastive representation learning uses in-batch negatives, treating other examples in the minibatch as the negative phase. Reward models in RLHF raise the score of a preferred response and lower the score of a rejected one, which is a two-sample energy update with human-supplied negatives.
Direct preference optimisation is worth naming here explicitly: its objective is exactly an energy-based contrastive update where the negative sample is provided by annotation rather than by a sampler. Recognising this is why the reward-hacking literature and the energy-based literature describe the same failure in different vocabularies.
When it breaks
Negative sample quality determines everything. The model only learns to lower its assignment of probability to regions its negatives actually reach. Easy negatives teach nothing, which is why hard negative mining matters so much in contrastive retrieval training and why it is the same problem as a poorly mixing chain.
Short chains hide the failure. With \(k=1\) the negatives look like the data, the loss goes down smoothly, and the model can be assigning high probability to enormous regions of nonsense without any training signal reporting it. Evaluating by sampling with a long chain, rather than by the training loss, is the check that reveals it.
Persistent chains fail silently after a learning-rate change. Increasing the learning rate mid-training can push the parameters faster than the chain pool can follow, so the negatives become stale. The symptom is training that looks fine and samples that degrade.
Regularisation is doing more work than it appears. Energy magnitude penalties, gradient clipping, and noise injection into the chain are usually presented as stabilisation details. In practice they are what keeps an under-constrained negative phase from letting the energy surface diverge, and removing them does not merely slow training, it breaks it.
12 flashcards for this concept
Click a card to reveal the answer.