Posterior Collapse and the KL Term

The KL term charges a cost for every nat the encoder puts into \(z\), and it is exactly zero when \(q(z \mid x) = p(z)\) for all \(x\). The model pays that cost only if the decoder converts the information into a larger reconstruction gain. A strong autoregressive decoder models \(p(x)\) well without any latent, so the gain is small, the cost is not, and ignoring \(z\) is the better point on the ELBO.

Because it cannot model correlations between output dimensions on its own. A factorised Gaussian over pixels has no way to make neighbouring pixels agree except through the latent, so the reconstruction gain from an informative \(z\) is large and easily outweighs the KL cost. Collapse is therefore a function of decoder capacity, which is why it appeared first and worst in text VAEs.

Free bits removes the incentive permanently rather than deferring it. Taking \(\max(\lambda, D_{\mathrm{KL},i})\) per dimension means the first \(\lambda\) nats in each dimension are unpenalised, so there is never a gradient pushing a dimension to exactly zero. Annealing only delays the full cost, so a schedule that ramps too fast collapses anyway. The price of free bits is that the loss is no longer a valid likelihood bound.

The direction and the goal. Annealing raises the KL coefficient from 0 toward 1 to prevent collapse and preserve reconstruction. Beta-VAE pushes it above 1 to deliberately increase pressure, buying more prior-like and more disentangled latents at the cost of reconstruction. There is no universally right weight, because the right one depends on whether you want a generative model or a representation.

Reconstruction and KL separately, and KL per latent dimension. A healthy model shows a few nats per active dimension and a visible split between active and inactive ones; a collapsed model is near zero across the board. Mutual information between \(x\) and \(z\), estimable from a minibatch, is the direct measurement and belongs on the dashboard rather than in an occasional audit.

Shuffle \(z\) across the batch and re-decode. If reconstruction quality barely changes, the decoder is ignoring the latent even though the encoder is paying KL to write into it. This catches the failure mode that per-dimension KL cannot see: an informative code that no downstream component reads.

Because the model is free to spend those nats on whatever most reduces reconstruction error, which is usually low-level detail such as exact texture or colour, not the semantic factors you wanted. Forcing the latent to carry information says nothing about which information. Disentanglement objectives and supervision are separate interventions from anti-collapse ones.

The KL term charges a cost for every nat the encoder puts into \(z\), and it is exactly zero when \(q(z \mid x) = p(z)\) for all \(x\). The model pays that cost only if the decoder converts the information into a larger reconstruction gain. A strong autoregressive decoder models \(p(x)\) well without any latent, so the gain is small, the cost is not, and ignoring \(z\) is the better point on the ELBO.

Because it cannot model correlations between output dimensions on its own. A factorised Gaussian over pixels has no way to make neighbouring pixels agree except through the latent, so the reconstruction gain from an informative \(z\) is large and easily outweighs the KL cost. Collapse is therefore a function of decoder capacity, which is why it appeared first and worst in text VAEs.

Free bits removes the incentive permanently rather than deferring it. Taking \(\max(\lambda, D_{\mathrm{KL},i})\) per dimension means the first \(\lambda\) nats in each dimension are unpenalised, so there is never a gradient pushing a dimension to exactly zero. Annealing only delays the full cost, so a schedule that ramps too fast collapses anyway. The price of free bits is that the loss is no longer a valid likelihood bound.

The direction and the goal. Annealing raises the KL coefficient from 0 toward 1 to prevent collapse and preserve reconstruction. Beta-VAE pushes it above 1 to deliberately increase pressure, buying more prior-like and more disentangled latents at the cost of reconstruction. There is no universally right weight, because the right one depends on whether you want a generative model or a representation.

Reconstruction and KL separately, and KL per latent dimension. A healthy model shows a few nats per active dimension and a visible split between active and inactive ones; a collapsed model is near zero across the board. Mutual information between \(x\) and \(z\), estimable from a minibatch, is the direct measurement and belongs on the dashboard rather than in an occasional audit.

Shuffle \(z\) across the batch and re-decode. If reconstruction quality barely changes, the decoder is ignoring the latent even though the encoder is paying KL to write into it. This catches the failure mode that per-dimension KL cannot see: an informative code that no downstream component reads.

Because the model is free to spend those nats on whatever most reduces reconstruction error, which is usually low-level detail such as exact texture or colour, not the semantic factors you wanted. Forcing the latent to carry information says nothing about which information. Disentanglement objectives and supervision are separate interventions from anti-collapse ones.