The Four Faces of KL Divergence: Mode-Seeking, Mode-Covering, and Why Your Estimator Went Negative

You add a KL penalty to an RLHF objective, log its value every step, and somewhere in the first hundred steps it prints \(-0.031\). KL divergence is non-negative — that is not a convention, it is a theorem, provable in one line from Jensen's inequality. Your training loop is printing a negative value for a quantity that cannot be negative, and there is no bug.

The explanation, worked out precisely later in this article, is that the standard sampled estimator of KL is negative slightly more than half the time on a representative example, while still being exactly unbiased. This is not a curiosity. It is the fourth of four distinct things "KL divergence" means in a modern machine learning system, and the four are routinely conflated:

1. The forward direction, \(D_{\mathrm{KL}}(p \,\Vert \, q)\), which every maximum-likelihood objective minimises and which forces \(q\) to spread out.
2. The reverse direction, \(D_{\mathrm{KL}}(q \,\Vert \, p)\), which every variational objective minimises and which forces \(q\) to concentrate.
3. KL as a local geometry, whose second-order expansion is the Fisher information and which turns up wherever a trust region does.
4. KL as an estimation problem, where you have samples rather than densities and the estimator's behaviour is its own subject.

They are the same formula. They behave like four different tools.

Why this matters: Which direction you write down decides whether your model hedges or commits. It explains why distillation produces a student that covers the teacher's alternatives, why variational autoencoders produce blurry samples, why an RLHF policy collapses toward one style, and why the number in your logs disagrees with the theorem. These are not four separate phenomena; they are one asymmetry, seen from four angles.

TL;DR

• KL is not a distance. It is asymmetric and violates the triangle inequality, so \(D_{\mathrm{KL}}(p\Vert q)\) and \(D_{\mathrm{KL}}(q\Vert p)\) are different objectives with different optima, not two ways of writing the same thing.
• Forward KL is zero-avoiding. Minimising \(D_{\mathrm{KL}}(p\Vert q)\) punishes \(q\) for putting near-zero mass where \(p\) has mass, so the optimum spreads across all of \(p\)'s modes, including the space between them.
• Reverse KL is zero-forcing. Minimising \(D_{\mathrm{KL}}(q\Vert p)\) costs nothing where \(q\) itself is zero, so the optimum picks a mode and ignores the rest.
• The worked example makes them choose opposite answers. For one bimodal target and two candidates, forward KL prefers the broad candidate at 0.595 against 1.234, while reverse KL prefers the narrow one at 0.576 against 1.273.
• Cross-entropy training is forward KL. \(H(p, q) = H(p) + D_{\mathrm{KL}}(p\Vert q)\), and \(H(p)\) does not depend on the model, so every next-token-prediction run is a forward-KL minimisation, with all the mode-covering that implies.
• The ELBO is reverse KL. \(\log p(x) = \mathrm{ELBO} + D_{\mathrm{KL}}(q\Vert p(z\mid x))\), so maximising the ELBO minimises reverse KL — which is where the classic blurriness of variational models comes from.
• The naive estimator is negative about half the time. In the worked example a single sample of the standard \(k_1\) estimator is negative with probability 0.51, while being exactly unbiased; Schulman's \(k_3\) estimator is unbiased, always non-negative, and in that same example has roughly 26% of \(k_1\)'s variance.
• Estimator choice is still an active research area, with a 2025 Rao-Blackwellised estimator proven to have variance no greater than the standard Monte Carlo one.

At a Glance

flowchart LR
  A["One formula: sum p log p over q"] --> B{Which argument do you optimise?}
  B -->|"Optimise q in KL(p to q)"| C["Forward: zero-avoiding"]
  B -->|"Optimise q in KL(q to p)"| D["Reverse: zero-forcing"]
  A --> E{Or is it a constraint?}
  E -->|"Small perturbation"| F["Local geometry: Fisher information"]
  A --> G{Do you have densities or samples?}
  G -->|"Samples only"| H["Estimation problem: k1, k2, k3"]
  C --> I["Maximum likelihood, distillation, cross-entropy"]
  D --> J["Variational inference, ELBO, RLHF-style objectives"]
  F --> K["Trust regions, natural gradient"]
  H --> L["Logged KL, penalty coefficients, early stopping"]
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  class A blue
  class B,E,G purple
  class C,I emerald
  class D,J rose
  class F,K amber
  class H,L blue

[IMAGE: A single bimodal target density drawn in grey, with two fitted Gaussians overlaid: one wide, centred in the valley between the modes, labelled "forward KL optimum"; one narrow, sitting on the taller mode, labelled "reverse KL optimum". Caption: "Same target, same family of candidates, opposite answers — the asymmetry is the whole story."]

Where It Came From

Relative entropy falls straight out of Shannon's 1948 framework: if you build an optimal code for distribution \(q\) and then send data drawn from \(p\), the expected number of extra bits per symbol is exactly \(D_{\mathrm{KL}}(p\Vert q)\). That interpretation — excess description length from using the wrong model — is the one worth keeping, because it explains the asymmetry immediately. The penalty is charged over samples from \(p\), so it depends on where \(p\) puts its mass, not where \(q\) does.

timeline
    title One quantity, four careers
    1948 : Shannon formalises entropy and coding
         : Relative entropy appears as the cost of coding with the wrong distribution
    1951 : Kullback and Leibler publish the directed divergence and its properties
    2013 : Kingma and Welling put reverse KL at the centre of the variational autoencoder
    2015 : Hinton, Vinyals and Dean use forward KL to a teacher for distillation
         : Schulman et al. use KL as a trust region constraint in TRPO
    2020 : Schulman circulates the k1, k2, k3 analysis of sampled KL estimators
    2022 : RLHF ships a KL penalty against a frozen reference policy at production scale
    2025 : Amini, Vieira and Cotterell prove a Rao-Blackwellised estimator dominates Monte Carlo

The Formula and Its One Real Property

For discrete distributions,

Non-negativity follows from Jensen's inequality applied to the convex function \(-\log\):

with equality if and only if \(p = q\) almost everywhere. Two structural facts follow from the definition and matter constantly. The expectation is over \(p\), which is why the two directions differ. And if there is any \(x\) with \(p(x) > 0\) but \(q(x) = 0\), the divergence is infinite — the absolute continuity requirement, which is exactly why every practical implementation clamps, smooths, or adds \(\epsilon\), and why KL is useless between distributions with disjoint support (the motivation for optimal transport; see optimal transport and Wasserstein).

Face One: Forward KL, and Why Your Language Model Hedges

Minimise \(D_{\mathrm{KL}}(p\Vert q)\) over \(q\), with \(p\) the data distribution. Expand:

The second term is the cross-entropy. Since \(H(p)\) does not depend on the model, minimising cross-entropy is minimising forward KL, which means every next-token-prediction run ever performed is a forward-KL minimisation (see cross-entropy and KL).

The behavioural consequence is in the weighting. The penalty \(p(x)\log\frac{p(x)}{q(x)}\) is charged wherever \(p\) has mass, and it grows without bound as \(q(x) \to 0\) there. So the optimiser will do almost anything to avoid assigning near-zero probability to something that actually occurs. It is zero-avoiding, or mode-covering: given a bimodal target and a unimodal family, the forward-KL optimum straddles both modes and places substantial mass in the valley between them, where the data never falls.

That is the formal version of a familiar complaint. A model trained by maximum likelihood hedges, because hedging is what its loss rewards; the loss is fully satisfied by covering every observed possibility and is indifferent to mass placed where nothing was observed.

[IMAGE: A four-bar discrete target with two tall outer bars and two near-zero inner bars, overlaid with a uniform candidate. Above each inner bar, an annotation shows the per-term forward-KL cost exploding as the candidate's probability there approaches zero. Caption: "Forward KL charges an unbounded price for assigning near-zero probability to something that happens."]

Face Two: Reverse KL, and Why Your Fine-Tune Collapses

Now minimise \(D_{\mathrm{KL}}(q\Vert p)\), with the expectation taken over the model's own samples:

Where \(q(x) = 0\) the term vanishes — \(\lim_{t\to 0} t\log t = 0\) — no matter how much mass \(p\) has there. Ignoring a whole mode of the target is free. What is not free is placing mass where \(p\) has none, since that drives the ratio up without bound. So reverse KL is zero-forcing, or mode-seeking: it finds one region of high \(p\) and concentrates there.

This is the direction in variational inference, because it is the one you can compute. The evidence lower bound decomposition,

shows that maximising the ELBO is exactly minimising the reverse KL to the true posterior, and the reason it is usable is that reverse KL requires expectations under \(q\), which you can sample, rather than under the intractable posterior (Kingma & Welling, 2013, Auto-Encoding Variational Bayes, arXiv:1312.6114). Variational posteriors are famously over-confident, and the ELBO's own structure is why.

It is also the direction implied by many RL-style objectives on language models, where the expectation is over the policy's samples. A policy optimised against a reward with a reverse-KL flavour will narrow toward the highest-reward mode — which is desirable when you want a reliable assistant and undesirable when it collapses the diversity you needed.

flowchart TB
  subgraph FWD["Forward: minimise KL(p to q)"]
    F1["Expectation over the data p"] --> F2["Blows up where p is positive and q is near zero"]
    F2 --> F3["Zero-avoiding: cover every mode"]
    F3 --> F4["Maximum likelihood, cross-entropy, distillation"]
  end
  subgraph REV["Reverse: minimise KL(q to p)"]
    R1["Expectation over the model q"] --> R2["Costs nothing where q is zero"]
    R2 --> R3["Zero-forcing: pick one mode"]
    R3 --> R4["ELBO, variational inference, policy objectives"]
  end
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  class F1,F2,F3,F4 emerald
  class R1,R2,R3,R4 rose

Face Three: KL as Local Geometry

Take a parametric family \(p_\theta\) and perturb slightly. The first-order term of the Taylor expansion of \(D_{\mathrm{KL}}(p_\theta \Vert p_{\theta + \delta})\) vanishes — the divergence is minimised at \(\delta = 0\) — so the leading behaviour is quadratic:

where \(F\) is the Fisher information matrix. Locally, KL is a squared distance, in a metric determined by the model rather than by the parameterisation. That is the entire content of natural gradient methods: descend in the direction that changes the distribution by a fixed amount, not the one that changes the parameters by a fixed amount (see Fisher information and natural gradient).

[IMAGE: A two-dimensional parameter space with concentric circles marking equal Euclidean distance, overlaid with distorted ellipses marking equal KL divergence, elongated along a direction the model is insensitive to. Caption: "Equal steps in parameters are not equal steps in behaviour; the Fisher metric is the difference."]

It also explains why KL rather than parameter norm is the right constraint for a policy update. A tiny change in weights can be an enormous change in behaviour and vice versa; the quantity you want to bound is the behavioural one. TRPO makes this literal, constraining average KL between successive policies (Schulman et al., 2015, arXiv:1502.05477).

Note that the symmetry of the quadratic form is a local fact. To second order the two directions agree, which is why the trust-region use of KL is comparatively insensitive to which direction you write — and why intuitions transferred from that setting mislead badly when applied to the global objectives of faces one and two.

stateDiagram-v2
    [*] --> Reference
    Reference: Freeze the reference policy pi_ref
    Reference --> Sample
    Sample: Generate responses from pi_theta
    Sample --> Estimate
    Estimate: Estimate KL(pi_theta to pi_ref) from those samples
    Estimate --> Penalise
    Penalise: Subtract beta times the estimate from the reward
    Penalise --> Update
    Update: Policy gradient step
    Update --> Sample: KL within budget
    Update --> Retune: KL drifting up despite the penalty
    Retune: Raise beta, or stop early
    Retune --> Sample
    Update --> [*]: training budget exhausted

The loop shows where face four enters and why it matters operationally: the Estimate box is the only place the KL is ever a number, and every decision downstream — the penalty magnitude, the drift alarm, the early-stopping rule — is made from that number.

Face Four: You Cannot Compute It, Only Estimate It

For two language models you have neither density in closed form over sequences; you have samples and per-sample log-probabilities. Writing \(r = q(x)/p(x)\) with \(x \sim p\), the obvious estimator is

Unbiased, and badly behaved: \(\log r\) is positive whenever \(q(x) > p(x)\), which happens for a large fraction of samples, and each such sample contributes a negative estimate of a non-negative quantity. Schulman's analysis lays out two alternatives (Schulman, 2020, Approximating KL Divergence):

\(k_2\) is always non-negative and biased. \(k_3\) is the interesting one. It is non-negative for every \(r > 0\), since \(r - 1 - \log r \ge 0\) with equality only at \(r = 1\), and it is also unbiased, because \(\mathbb{E}_p[r] = \sum_x p \cdot \frac{q}{p} = 1\), so the added term \((r-1)\) contributes exactly zero in expectation. It is a control variate: a quantity with known mean zero, correlated with the estimator, subtracted to reduce variance for free. Getting non-negativity and unbiasedness together, at no cost, is why \(k_3\) is the default in modern RLHF implementations.

The subject did not stop there. A 2025 result introduces a Rao-Blackwellised estimator that is unbiased and provably has variance less than or equal to the standard Monte Carlo estimator, reporting more stable KL estimates and better reward-versus-KL frontiers when used for the gradient as well (Amini, Vieira & Cotterell, 2025, Better Estimation of the Kullback-Leibler Divergence Between Language Models, NeurIPS, arXiv:2504.10637).

Watch It Run

By the Numbers

All values below are computed exactly from the worked example that follows, using natural logarithms, and are reproducible with a pocket calculator.

Sources: the divergence and estimator values are computed from the four-outcome distributions specified below. The \(k_1\) / \(k_2\) / \(k_3\) estimator family is from Schulman (2020); the claim that a Rao-Blackwellised estimator provably dominates Monte Carlo is from Amini et al. (2025).

A Concrete Example

Take four outcomes and a sharply bimodal target:

and two candidate approximations — a broad one and a narrow one sitting on a single mode:

Step 1: forward KL for the broad candidate. By symmetry the two outer terms are equal and the two inner terms are equal:

Step 2: forward KL for the narrow candidate.

The dominant term is \(0.49 \log(0.49/0.02) = 1.567\): the narrow candidate assigns 2% probability to an outcome that occurs 49% of the time, and forward KL charges heavily for it. Forward KL prefers \(q_1\), 0.595 against 1.234.

Step 3: reverse KL, same two candidates.

The broad candidate is now the expensive one, because it puts 0.25 probability on outcomes the target gives 0.01 — the term \(2(0.25)\log 25 = 1.609\) dominates everything. Reverse KL prefers \(q_2\), 0.576 against 1.273.

Two objectives, one target, one candidate set, and the rankings are exactly reversed. If you had swapped the argument order by accident, you would not get a slightly worse answer; you would get the other answer.

Step 4: now estimate \(D_{\mathrm{KL}}(p\Vert q_2)\) from samples. Draw \(x \sim p\) and compute \(r = q_2(x)/p(x)\) for each outcome:

Step 5: check both estimators.

Both are exactly unbiased. Now look at the third column of the table again. Outcomes 2, 3 and 4 give a negative \(k_1\), and their combined probability is \(0.01 + 0.01 + 0.49 = 0.51\). A single-sample estimate of this KL divergence is negative 51% of the time, which is the log line from the opening paragraph, derived rather than observed.

Step 6: compare variances. Computing second moments and subtracting \(1.234^2 = 1.523\):

\(k_3\) carries about 26% of \(k_1\)'s variance and is never negative, for the cost of adding \((r-1)\) — a term whose expectation is provably zero. That is the whole argument for control variates in one line of arithmetic.

[IMAGE: Two panels. Left, a bar chart of the four per-outcome k1 values with three bars below zero shaded red and one large positive bar shaded green, annotated "51% of samples land below zero". Right, the same four outcomes for k3, all bars above zero and visibly shorter. Caption: "Same expectation, one quarter the variance, no impossible values."]

Where It Breaks

The direction is decided by what you can sample, not by what you want

This is the most under-appreciated constraint in the whole subject. Forward KL needs expectations over the target, so you can only use it when you can sample the target — which is why supervised learning uses it and variational inference cannot. Reverse KL needs expectations over the model, which you can always sample. The prevalence of reverse KL in generative modelling is largely a statement about tractability, and its mode-seeking behaviour is a side effect that then has to be managed. Treating the choice as a modelling decision, when it was made for you by what is samplable, leads to a long search for the wrong fix.

Infinite divergence and the epsilon that hides it

If \(q\) assigns zero probability where \(p\) does not, the divergence is infinite, which is mathematically correct and operationally useless. Every implementation therefore clamps probabilities, adds \(\epsilon\), or smooths — and those choices silently determine the loss landscape near the boundary. Two implementations of "the same" KL penalty can behave differently for no reason other than their clamping.

Estimator bias in the gradient, not just the value

Most discussion of \(k_1\) versus \(k_3\) concerns the reported value, but in an RLHF loop the KL term is differentiated, and the gradient of an unbiased estimator is not automatically an unbiased gradient estimator. This is precisely the gap the Rao-Blackwellised work targets, reporting more stable training when the improved estimator is used for the gradient rather than only for logging. If you swapped \(k_1\) for \(k_3\) in your logging and left the loss untouched, you have improved your dashboard and not your training.

The RLHF penalty is not a trust region

A KL penalty against a frozen reference model and a KL trust region between successive policies are different mechanisms that share a name. The penalty bounds drift from the starting model and accumulates over training; the trust region bounds each step. A run can satisfy a small per-step KL at every step and still drift arbitrarily far from the reference, which is exactly how a policy quietly collapses while every logged step looks fine (see the KL penalty and reference model).

Second-order intuitions do not transfer to the global objectives

Because \(D_{\mathrm{KL}}(p_\theta\Vert p_{\theta+\delta})\) and its reverse agree to second order, people who first meet KL in a trust-region context absorb the impression that the direction is a minor detail. It is a minor detail there and a decisive one in faces one and two, where the distributions are far apart and the quadratic approximation is worthless.

KL is undefined where you often need it most

Two distributions with disjoint support have infinite KL, and — worse — the divergence carries no information about how far apart they are, so it provides no gradient signal to bring them together. This is the standard motivation for Wasserstein distances in generative modelling, and it is a genuine limitation rather than a technicality.

Alternative Designs

The \(\alpha\)-divergence row is the honest resolution of the forward-versus-reverse debate: they are two points on a continuum, and \(\alpha\) is a dial between covering and seeking. It is used less than it should be, mostly because the two endpoints have clean, cheap estimators and the interior does not.

[IMAGE: A horizontal alpha axis from the reverse-KL endpoint to the forward-KL endpoint, with a small fitted distribution shown at five points along it, morphing from a narrow spike on one mode to a broad hump covering both. Caption: "Mode-seeking and mode-covering are the two ends of one dial, not two philosophies."]

How It Is Used in Practice

Every language model is trained with forward KL. Next-token cross-entropy is forward KL to the empirical data distribution, and its mode-covering character is why base models assign non-trivial probability to many continuations rather than committing. Sampling temperature and nucleus filtering at inference are, in effect, post-hoc corrections to that hedging.

Distillation is forward KL to a teacher. Matching a teacher's full output distribution rather than its argmax is what transfers the "dark knowledge" of relative probabilities among wrong answers (Hinton, Vinyals & Dean, 2015, arXiv:1503.02531), and the student consequently inherits the teacher's hedging along with its knowledge. Teams that want a decisive student often find that reverse KL or a mixture works better, and the reason is exactly the asymmetry described here (see knowledge distillation).

RLHF uses KL as a leash and reports it as a health metric. The penalty coefficient \(\beta\) trades reward against drift, and the logged KL is the single most-watched number in a run, which is what makes face four operational rather than academic: a metric read off a high-variance estimator, and used to make stop-or-continue decisions, needs an estimator whose noise you understand.

[IMAGE: A reward-versus-KL scatter for an RLHF run, with each point a checkpoint, showing the frontier bending over as KL grows and points beyond a threshold losing reward. Two overlaid noise bands show the spread of the logged KL under the k1 and k3 estimators. Caption: "The stopping decision is made by reading a noisy estimate off this axis."]

Variational methods use reverse KL and pay for it. Blurry reconstructions and over-confident posteriors are ELBO artefacts, and the long list of fixes — importance-weighted bounds, normalising flows, richer posterior families — are all attempts to reduce the cost of the direction the tractability forced.

Insights Worth Remembering

1. KL is not a distance and the asymmetry is the feature. \(D_{\mathrm{KL}}(p\Vert q)\) measures the cost of using \(q\) to describe data from \(p\). Reversing the arguments asks a genuinely different question, and in the worked example the two questions have opposite answers.

2. The expectation's subscript tells you the behaviour. Averaging over \(p\) punishes \(q\) for missing \(p\)'s mass, producing mode-covering; averaging over \(q\) punishes \(q\) for straying outside \(p\), producing mode-seeking. Everything else follows from which distribution the sum runs over.

3. Direction is usually chosen by tractability, not by intent. Forward KL requires sampling the target; reverse KL requires sampling the model. When only one is possible, the mode behaviour is a consequence you must manage, not a decision you made.

4. Cross-entropy training and variational inference are the two faces, and they explain opposite pathologies. Hedging language models and over-confident posteriors are the same asymmetry seen from either side.

5. Locally, KL is the Fisher metric. That is why it is the right thing to constrain in a policy update and why natural gradient exists. It is also why intuitions from trust regions mislead when carried to global objectives, where the quadratic approximation does not hold.

6. A negative logged KL is an estimator artefact, not a bug. In the worked example a single \(k_1\) sample is negative 51% of the time while being exactly unbiased. If a value that cannot be negative is negative, suspect the estimator before the code.

7. \(k_3\) is a control variate and it is free. Adding \((r-1)\), whose expectation is exactly zero, buys non-negativity and roughly a four-fold variance reduction in the worked example. Free variance reduction is rare enough that when you see it you should take it.

8. Estimating KL well is still an open research problem. A 2025 Rao-Blackwellised estimator with provably lower variance, applied to the gradient rather than only to logging, produced measurably more stable training — which means the quantity everyone watches has been imprecisely measured for years.

Open Questions

What is the right divergence for aligning language models? Reverse KL's mode-seeking is sometimes what you want, and sometimes it is the mechanism of diversity collapse. Whether a different divergence — an \(\alpha\)-interpolation, a symmetrised form, or something with an explicit diversity term — produces better-behaved policies at scale is an empirical question that has not been settled, partly because the estimators for the alternatives are less mature.

How much does estimator variance actually cost in RLHF? We now know the standard estimator is improvable and that the improvement helps on the Pareto frontier of reward against KL. How much of the instability practitioners attribute to reward hacking or to hyperparameters is instead a noisy KL estimate steering a control loop is not quantified.

Can the reference-drift problem be given a proper formulation? Penalising KL to a frozen reference is a blunt instrument: it treats every kind of drift as equally undesirable, when what is wanted is to preserve capability while allowing behaviour change. No accepted formalisation of that distinction exists, and the current answer is a scalar \(\beta\) tuned by eye.

Is the absolute continuity requirement fatal for high-dimensional generative modelling? Real data distributions plausibly live on low-dimensional manifolds, where the disjoint-support pathology is generic rather than exceptional. Whether KL-based objectives work in practice despite this, or because the noise and smoothing in real pipelines quietly repair it, is not clearly understood.

What is the right way to measure divergence between two language models? Divergence over sequences is what we want; per-token estimates are what we compute, and they aggregate in ways that depend on length, on tokenisation, and on where the distributions differ. A well-founded sequence-level measure that is cheap enough to log every step does not currently exist.

Sources and Further Reading

Foundations

1. Kullback, S., & Leibler, R. A. (1951). "On Information and Sufficiency." Annals of Mathematical Statistics, 22(1), 79–86. (Original formulation of the directed divergence; cited by venue rather than by link.)
2. Cover, T. M., & Thomas, J. A. (2006). Elements of Information Theory, 2^nd edition. Wiley. (Chapter 2 is the standard reference for non-negativity, the chain rule, and the coding interpretation.)

The two directions in practice

3. Kingma, D. P., & Welling, M. (2013). "Auto-Encoding Variational Bayes." arXiv:1312.6114
4. Hinton, G., Vinyals, O., & Dean, J. (2015). "Distilling the Knowledge in a Neural Network." arXiv:1503.02531

KL as geometry

5. Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015). "Trust Region Policy Optimization." ICML 2015. arXiv:1502.05477

Estimating KL

6. Schulman, J. (2020). "Approximating KL Divergence." joschu.net/blog/kl-approx.html
7. Amini, A., Vieira, T., & Cotterell, R. (2025). "Better Estimation of the Kullback-Leibler Divergence Between Language Models." NeurIPS 2025. arXiv:2504.10637

Related concepts on this site

8. Cross-entropy and KL
9. Probability and information theory
10. Fisher information and natural gradient
11. Optimal transport and Wasserstein
12. The KL-regularised RL objective
13. The KL penalty and reference model
14. Entropy and surprise

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.