Why Policy Gradients Need a Baseline: Variance, Trust Regions, and the Road to PPO

Take any reinforcement learning problem and add 1,000 to every reward. Nothing meaningful has changed: the same policy is optimal, the same actions are better than the same other actions, and the expected value of the REINFORCE gradient estimator is exactly what it was before. But the variance of the estimate you actually compute from samples explodes. In the three-action example worked out later in this article, the estimator's variance rises from about 24.5 to about 226,900 — a factor of roughly 9,250 — for a change that carries no information about which action is good.

That is not a pathological edge case. It is the defining property of the score-function estimator, and it says something uncomfortable: the quantity we are estimating is invariant to a transformation that the estimator is wildly sensitive to. The gap between those two facts is where every technique in this article lives. Baselines, advantage functions, critics, GAE's \(\lambda\), trust regions, and PPO's clipping are not a grab-bag of tricks; they are a sequence of answers to the question how do we get a usable gradient out of an estimator whose noise dwarfs its signal.

Why this matters: The bias-variance structure of the policy gradient estimator determines everything downstream — how many samples a run needs, whether it is stable, why identical hyperparameters give different results on different reward scales, and why the RLHF and RLVR algorithms training language models today look the way they do. Every one of them is a policy gradient method with a specific choice of baseline and a specific mechanism for bounding the update.

TL;DR

• The score-function estimator is unbiased and nearly unusable raw. \(\nabla_\theta J = \mathbb{E}[R \nabla_\theta \log \pi_\theta]\) is correct in expectation, and a single-sample estimate of it can point almost anywhere.
• Any baseline that does not depend on the action is free. Subtracting \(b(s)\) leaves the expectation exactly unchanged, because \(\mathbb{E}[\nabla_\theta \log \pi_\theta(a\mid s)] = 0\). This is the one genuinely free lunch in the whole field.
• In the worked example it cuts variance by a factor of about 331 with no bias whatsoever, and by roughly 3 million once the rewards carry a large constant offset.
• The advantage function is the baseline taken seriously. \(A(s,a) = Q(s,a) - V(s)\) replaces "how much return did I get" with "how much better than expected was this action", which is the quantity that should drive an update.
• Once you estimate the baseline, you buy bias. A learned critic is wrong early in training, and GAE's \(\lambda\) is the explicit dial between a high-bias one-step estimate and a high-variance Monte Carlo one.
• Trust regions solve a different problem. Variance reduction makes the direction reliable; trust regions stop you from walking too far along it, which matters because the surrogate objective is only locally valid.
• PPO's clipping is a heuristic that works, and the reasons are contested. A careful ablation found that code-level implementation details — reward normalisation, value clipping, orthogonal initialisation, learning-rate annealing — account for most of PPO's advantage over TRPO, not the clipped objective itself.
• GRPO removes the critic entirely by using the mean reward of a group of sampled responses as the baseline, which is the same 1992 idea recovered at LLM scale.

At a Glance

flowchart LR
  A[Sample trajectories from the current policy] --> B["Score function: grad log pi"]
  B --> C[Weight by return]
  C --> D{Subtract a baseline?}
  D -->|"No: REINFORCE"| E["Very high variance, unbiased"]
  D -->|"Yes: advantage"| F["Low variance, still unbiased if b is action-independent"]
  F --> G{How is the baseline estimated?}
  G -->|"Learned critic"| H[Bias from critic error]
  G -->|"Group mean of samples"| I[Bias from group size]
  H --> J["Bound the step: trust region or clipping"]
  I --> J
  E --> J
  J --> K[Policy update]
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  class A,B,C blue
  class D,G purple
  class E rose
  class F emerald
  class H,I amber
  class J,K purple

The two diamonds are the whole design space. The first is free; the second is not.

[IMAGE: Two scatter plots of single-sample gradient estimates in a two-dimensional parameter space, with the true gradient drawn as a black arrow. Left, without a baseline: a wide cloud of points spanning all directions, mean coinciding with the arrow. Right, with a baseline: a tight cluster around the same arrow. Caption: "Same expectation, same arrow, radically different sample."]

Before Trust Regions: Thirty Years of the Same Fix

The estimator is old. Williams derived the REINFORCE class of algorithms in 1992 and — this is routinely forgotten — the original paper already includes a reinforcement baseline as part of the method (Williams, 1992, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning, Machine Learning 8, 229–256). The subsequent thirty years did not discover that baselines help; they worked out how to estimate good ones, and how to stop the resulting update from destroying the policy.

timeline
    title From likelihood ratios to clipped surrogates
    1992 : Williams derives REINFORCE
         : Baselines are present in the original formulation
    1999 : Sutton et al. prove the policy gradient theorem for function approximation
         : Konda and Tsitsiklis formalise actor-critic
    2002 : Kakade's natural policy gradient makes the update aware of the policy manifold's geometry
    2015 : Schulman et al. TRPO enforces a KL trust region with a monotonic improvement guarantee
         : Schulman et al. GAE exposes the bias-variance trade-off as a single parameter lambda
    2017 : Schulman et al. PPO replaces the constrained problem with a clipped surrogate
    2020 : Engstrom et al. show code-level details explain most of PPO's edge over TRPO
    2024 : GRPO drops the value network, using a group of sampled responses as the baseline

The Estimator, and Why the Baseline Is Free

Start from the objective \(J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)]\). The difficulty is that the distribution being averaged over depends on the parameters we are differentiating, so the gradient cannot pass through the expectation. The log-derivative identity \(\nabla_\theta p_\theta = p_\theta \nabla_\theta \log p_\theta\) fixes exactly that, turning the gradient of an expectation into an expectation of a gradient:

This is estimable from samples with no model of the environment, which is what made it possible at all. It is also, as an estimator, badly behaved: the return \(R(\tau)\) multiplies the entire score, so the magnitude of the update is driven by the scale of the reward rather than by how much better one action was than another.

The identity that makes baselines free

For any function \(b(s)\) that does not depend on the action:

The sum of the probabilities is 1, a constant, so its gradient vanishes. Subtracting \(b(s)\) from the return therefore changes the expectation by exactly zero while changing the variance by a great deal. Two conditions matter and are the source of most bugs in practice: \(b\) must not depend on the action, and it must not depend on the sample whose gradient it is scaling. Violate either and the estimator is quietly biased.

From baseline to advantage

The variance-minimising baseline is a score-weighted average return, but in practice everyone uses the state value \(V^\pi(s)\), which is close enough and interpretable. Doing so turns the weight on the score into

the advantage: how much better this action was than the policy's own average from this state. Sign now means something absolute — positive advantage means "do more of this" — where a raw return's sign meant only "the rewards in this environment happen to be positive".

[IMAGE: A single state with four action branches, annotated with returns 98, 101, 99, 102 and the state value 100 drawn as a horizontal reference line. Beneath, the same branches labelled with advantages minus 2, plus 1, minus 1, plus 2. Caption: "The information was always the spread, never the level."]

Estimating the Baseline: GAE and the Bias Dial

You do not have \(V^\pi\); you have a network approximating it, and that is where the free lunch ends. Two extreme estimators of the advantage bracket the space:

The one-step form has low variance — it touches one reward — and high bias, because it inherits every error in \(V\). The Monte Carlo form is nearly unbiased and has variance accumulating over the whole trajectory. Generalised advantage estimation interpolates between them with an exponentially weighted sum of the TD residuals \(\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)\) (Schulman et al., 2015, High-Dimensional Continuous Control Using Generalized Advantage Estimation, arXiv:1506.02438):

At \(\lambda = 0\) this collapses to the one-step residual; at \(\lambda = 1\) it recovers the Monte Carlo advantage. The parameter is a single, legible dial on how much you trust your critic, and it is exactly analogous to \(\text{TD}(\lambda)\) in value-based learning. In practice \(\lambda\) is set close to but below 1, and that choice is the admission that the critic is useful but not to be trusted at long horizons.

[IMAGE: A horizontal dial from lambda equals 0 to lambda equals 1, with a bias curve falling and a variance curve rising across it, crossing near the right-hand side; annotations mark "trusts the critic completely" at the left end and "ignores the critic, pure Monte Carlo" at the right. Caption: "One parameter, one honest question: how wrong is your value function?"]

Note that \(\gamma\) has quietly acquired a second job. It is nominally the discount factor defining the problem, and it now also acts as a variance-reduction parameter, downweighting distant rewards whose credit assignment is unreliable. These two roles are conceptually distinct and are almost always conflated in a single hyperparameter (see returns, discounting and episodes).

Bounding the Step: Trust Regions and Clipping

Reducing variance makes the direction trustworthy. It says nothing about distance, and distance is a separate hazard: the surrogate objective policy gradients optimise is a local approximation, valid near the current policy, and a large step lands somewhere the approximation never described. In supervised learning a too-large step costs you a bad update. In RL it costs you the data distribution, because the damaged policy collects the trajectories that produce the next update, and there is no fixed dataset to recover from.

TRPO addresses this directly, maximising a surrogate objective subject to a hard constraint on the average KL divergence between the old and new policies, and comes with a monotonic improvement guarantee for the idealised version (Schulman et al., 2015, Trust Region Policy Optimization, arXiv:1502.05477). The price is the machinery: a constrained optimisation solved by conjugate gradient against Fisher-vector products, with a line search to enforce the constraint.

PPO replaces all of that with a clip (Schulman et al., 2017, arXiv:1707.06347). With the importance ratio \(r_t(\theta) = \pi_\theta(a_t\mid s_t)/\pi_{\theta_{\text{old}}}(a_t \mid s_t)\):

The min is doing something subtler than it looks. When the advantage is positive, the clipped term caps the incentive to increase the probability further, so improvement past the trust region earns nothing. When the advantage is negative, the same expression leaves the decrease uncapped, because reducing the probability of a bad action is safe. It is a one-sided pessimistic bound, not a symmetric constraint, and it enforces nothing: a single step can move the ratio far outside \([1-\epsilon, 1+\epsilon]\); it simply gains no surrogate reward for doing so. PPO does not implement a trust region. It removes the incentive to leave one.

stateDiagram-v2
    [*] --> Collect
    Collect: Roll out N steps with the frozen old policy
    Collect --> Estimate
    Estimate: Compute GAE advantages and value targets
    Estimate --> Normalise
    Normalise: Standardise advantages within the batch
    Normalise --> Optimise
    Optimise: Several epochs of minibatch ascent on the clipped surrogate
    Optimise --> Check
    Check: Ratio drifted far from 1?
    Check --> Optimise: within range, keep going
    Check --> Collect: stale data, resample
    Collect --> [*]: budget exhausted

The loop makes the tension visible. Multiple epochs over the same batch is what makes PPO sample-efficient relative to a single-update method, and it is also what makes the data progressively off-policy as the ratio drifts, which is what the clip exists to survive.

Seeing It in Motion

flowchart TB
  subgraph R["REINFORCE"]
    R1[Full return R] --> R2[Times score]
    R2 --> R3["Unbiased, variance scales with reward level"]
  end
  subgraph B["REINFORCE with baseline"]
    B1["R minus b of s"] --> B2[Times score]
    B2 --> B3["Unbiased, variance scales with reward spread"]
  end
  subgraph AC["Actor-critic and GAE"]
    C1["Advantage from learned V"] --> C2[Times score]
    C2 --> C3["Biased by critic error, tuned by lambda"]
  end
  subgraph PP["PPO"]
    P1[GAE advantage] --> P2[Clipped importance ratio]
    P2 --> P3["Biased, bounded, reusable across epochs"]
  end
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  class R1,R2,R3 rose
  class B1,B2,B3 emerald
  class C1,C2,C3 amber
  class P1,P2,P3 purple

Reading left to right is reading the historical sequence, and each step trades a property away. Unbiasedness goes first, and it goes cheaply, because an unbiased estimator you cannot afford to average enough times is worse than a slightly biased one you can.

Watch It Run

By the Numbers

The variance figures below are computed exactly from the three-action example in the next section, not measured or cited; they are reproducible in a few lines of arithmetic, and that is the point of choosing a tiny problem.

Every row has the same expected gradient. The offset in rows three and four carries no information about which action is better, and it inflates the unbaselined estimator's variance by a factor of roughly 9,250 relative to row one.

The structural comparison of the algorithms is a matter of record rather than measurement:

Sources: TRPO (Schulman et al., 2015), PPO (Schulman et al., 2017), GRPO as introduced in DeepSeekMath (Shao et al., 2024, arXiv:2402.03300).

A Concrete Example

A single-state bandit with three actions, a softmax policy at uniform initialisation \(\pi = (\tfrac13, \tfrac13, \tfrac13)\), and rewards \(r = (10, 11, 12)\).

Step 1: write down the score. For a softmax policy the gradient of the log-probability with respect to the logits is \(\nabla_{\theta_i} \log \pi(a) = \mathbb{1}[i = a] - \pi_i\). So sampling action 1 gives the score vector \((\tfrac23, -\tfrac13, -\tfrac13)\), action 2 gives \((-\tfrac13, \tfrac23, -\tfrac13)\), and action 3 gives \((-\tfrac13, -\tfrac13, \tfrac23)\).

Step 2: compute the true gradient. The estimator is \(g = r(a) \cdot \text{score}(a)\), and its expectation over the uniform policy, taking the first component:

Doing the same for the other components gives \(\mathbb{E}[g] = (-\tfrac13, 0, \tfrac13)\): push probability away from action 1 and toward action 3, which is right.

Step 3: compute the variance without a baseline. The three possible values of \(g_1\) are \(10 \cdot \tfrac23 = 6.667\), \(11 \cdot (-\tfrac13) = -3.667\), and \(12 \cdot (-\tfrac13) = -4\). Then

so \(\operatorname{Var}[g_1] = 24.63 - (\tfrac13)^2 = 24.52\). The standard deviation is about 4.95 against a mean of −0.333: the noise is roughly fifteen times the signal, from a problem with three actions and no time dimension at all.

Step 4: subtract the mean reward as a baseline. With \(b = 11\) the advantages become \((-1, 0, 1)\) and the three values of \(g_1\) are \(-\tfrac23\), \(0\), and \(-\tfrac13\). The expectation is \(\tfrac13(-\tfrac23 + 0 - \tfrac13) = -\tfrac13\) — unchanged, as the identity promised. The second moment is \(\tfrac13(\tfrac49 + 0 + \tfrac19) = 0.1852\), so

a variance reduction of \(24.52 / 0.0741 \approx 331\times\) for no bias and essentially no compute.

Step 5: shift the rewards. Replace \(r\) with \((1010, 1011, 1012)\). The optimal action is unchanged; the expected gradient is unchanged at \(-\tfrac13\); and the three values of \(g_1\) become \(673.33\), \(-337\), and \(-337.33\), giving \(\mathbb{E}[g_1^2] = 226{,}913.6\) and a variance of about \(226{,}913\). That is 9,250 times the variance of step 3, produced entirely by a constant that means nothing.

With the baseline \(b = 1011\), the advantages are \((-1, 0, 1)\) again and the variance is exactly \(0.0741\), identical to step 4. The baselined estimator is invariant to the shift; the raw one is not.

This is the entire argument for baselines in eight lines of arithmetic, and it generalises directly: in a real environment the "constant offset" is whatever portion of the return is explained by being in a good state rather than by taking a good action, and \(V(s)\) is the estimate of precisely that portion.

[IMAGE: Bar chart on a log-scaled y-axis with four bars matching the By the Numbers table rows, annotated with the identical expected gradient on each. Caption: "Four estimators of the same quantity, spanning seven orders of magnitude in variance."]

Where It Breaks

Action-dependent baselines are silently biased

The identity that makes baselines free requires \(b\) to be independent of the sampled action. It is easy to violate accidentally: normalising advantages using statistics that include the current sample, using a critic that was updated on this batch before computing advantages from it, or a state-action baseline adopted without the corrective term. The failure is not a crash but a slow drift toward a wrong optimum, and it is invisible without a controlled comparison.

Batch advantage normalisation is a bias you have already accepted

Standardising advantages within a minibatch is near-universal and it is not covered by the identity: the mean and standard deviation are functions of the sampled actions. Empirically it helps enough that everyone does it. Formally it makes the estimator biased. This is worth stating plainly because it is the most common gap between what practitioners do and what the derivation licenses.

The critic and the policy are a moving-target pair

The advantage depends on a critic trained on data generated by a policy that is being changed by the advantage. Early in training the critic is wrong, so advantages are wrong, so the policy moves badly, so the critic's training data is bad. Value-loss coefficients, separate learning rates, and shared-versus-separate trunks are all attempts to manage this coupling, and none of them removes it.

PPO does not enforce what it appears to enforce

The clipped objective removes the incentive to move far but does not constrain the step, so a large-enough learning rate produces ratios far outside the clip range. More pointedly, an ablation study attributed most of PPO's improvement over TRPO to code-level optimisations — reward and observation normalisation, value-function clipping, orthogonal initialisation, learning-rate annealing, gradient clipping — rather than to the clipped surrogate itself, and found these details substantially change agent behaviour (Engstrom et al., 2020, Implementation Matters in Deep Policy Gradients, ICLR, arXiv:2005.12729). Reproducing "PPO" without those details reproduces a different algorithm.

[IMAGE: Two plots of the clipped surrogate objective against the importance ratio r, one for positive advantage and one for negative. The positive-advantage curve flattens above 1 plus epsilon; the negative-advantage curve keeps descending below 1 minus epsilon without flattening. Caption: "The clip is one-sided: capped upside for good actions, uncapped downside for bad ones."]

Reward scale is a hyperparameter you did not know you had

Everything above is a statement about scale sensitivity, and it explains a common experience: an algorithm that works on one environment fails on another whose rewards are ten times larger, with the same hyperparameters. Reward normalisation is the standard mitigation and interacts with discounting, with advantage normalisation, and with the value loss in ways that are rarely disentangled.

Group baselines have their own bias

GRPO's baseline is the mean reward over a group of sampled responses to the same prompt, which removes the value network at the cost of a baseline computed from the same samples it scales — and a group of modest size gives a noisy estimate. It also changes what "advantage" means: relative to this prompt's sampled group, not to the policy's value function, which makes prompts with uniformly high or uniformly low reward contribute almost nothing (see the KL-regularised RL objective for the other half of the LLM-era objective).

Alternative Designs

The historical direction is consistent: give up theoretical guarantees for implementability, then recover empirical performance through engineering. TRPO has a monotonic improvement guarantee that PPO lacks, and PPO is what people run.

[IMAGE: A two-axis chart with theoretical guarantee strength on one axis and implementation simplicity on the other, plotting REINFORCE, natural policy gradient, TRPO, PPO and GRPO. Caption: "The field has moved steadily down-right, and empirical performance has moved with it."]

How It Is Used in Practice

Every RLHF pipeline is this article's machinery with a language model in the policy slot. A prompt is a state, a generated response is an action or a sequence of them, a reward model supplies the return, and PPO with GAE performs the update — plus a KL penalty against the frozen initial policy that is a different mechanism from the trust region, constraining drift from the starting model rather than from the previous iterate (see the KL penalty and reference model).

[IMAGE: An RLHF pipeline diagram annotated with this article's vocabulary: prompt as state, sampled response as action, reward model output as return, value head as baseline, GAE as advantage estimator, clip as step control, and a separate KL-to-reference arrow drawn in a different colour to mark that it is not the trust region. Caption: "The same five components, renamed."]

Three things about the LLM setting change the calculus. Rewards are usually sequence-level, so credit assignment across hundreds of tokens is done by the critic with almost no signal to work from — which is much of why critic-free methods took hold. The value network is a second model of comparable size, so removing it is a memory saving measured in tens of gigabytes rather than a theoretical convenience. And with verifiable rewards, the reward is exact rather than a learned approximation, which removes reward-model overoptimisation and leaves the variance problem exactly as it was (see RL from verifiable rewards).

In classical control the pattern is different: PPO with a learned critic, GAE, normalised observations and rewards, and a great deal of attention to the implementation details listed above. Practitioners in this setting learn to distrust single-seed results, because the variance discussed throughout this article shows up as run-to-run spread large enough to reverse the apparent ranking of two algorithms (see error bars for LLM evals for the same statistical hygiene in another setting).

Insights Worth Remembering

1. The baseline identity is the only free lunch in policy gradients. Subtracting any action-independent function leaves the gradient's expectation exactly unchanged and can cut variance by orders of magnitude. Everything else in this article buys variance reduction with bias.

2. Reward scale is not a cosmetic property of an environment. The unbaselined estimator's variance grows with the level of the rewards, not their spread, so a constant offset that changes nothing about the problem can change everything about learnability. If an algorithm behaves differently after adding a constant to the reward, that is a bug in the estimator, not in the environment.

3. The advantage function is the answer to "compared to what?". A return says how well things went; an advantage says how much of that was attributable to the action. Only the second is a valid learning signal, and \(V(s)\) exists to separate them.

4. \(\lambda\) in GAE is a statement about how much you trust your critic. Near zero says the critic is good and the returns are noisy; near one says the opposite. It is one of the few hyperparameters in deep RL with a clean interpretation, and it should be set by reasoning about the critic rather than by grid search alone.

5. Variance control and step control are different problems with different solutions. Baselines and critics make the direction reliable; trust regions and clipping bound the distance. Confusing them leads to tuning the wrong knob when a run destabilises.

6. PPO's clip is one-sided by design. It caps the reward for increasing the probability of a good action, and deliberately does not cap the decrease in probability of a bad one. That asymmetry is the mechanism, not an accident of the min.

7. Implementation details are the algorithm. The finding that normalisation, initialisation and annealing account for most of PPO's advantage over TRPO should change how any policy gradient result is read: the name of the method underdetermines what was run.

8. GRPO is Williams 1992 at scale. Using the mean reward of a sampled group as the baseline is the oldest idea in the field, made newly attractive because the alternative — a critic the size of the policy — is now expensive enough to be worth removing.

Open Questions

What is the right credit assignment unit for sequence-level rewards? With one scalar reward for a several-hundred-token response, per-token advantages are almost entirely critic invention, and critic-free methods assign the same advantage to every token in a response. Neither is right, and process supervision, step-level rewards, and learned segmenters are all active attempts at something better (see credit assignment over long generations).

Does clipping deserve its position, or did it merely arrive with good defaults? The Engstrom ablation is evidence that the clipped objective is not where PPO's advantage comes from. Whether a well-tuned method without clipping matches it under matched implementation details is still not cleanly established, because the comparison requires holding a dozen code-level choices fixed.

Can the critic be made reliable enough to pay for itself at LLM scale? Critic-free methods won partly on memory and partly because the critic was bad. If value estimation for long sequences improved substantially, the argument would need revisiting, and it is unclear whether the difficulty is fundamental or a matter of training recipe.

How should exploration interact with variance reduction? Entropy bonuses are the standard, crude tool, and they interact with advantage normalisation and clipping in ways that are tuned rather than understood (see entropy regularisation). A principled account of maintaining exploration under a bounded, low-variance update does not exist.

Is monotonic improvement recoverable at scale? TRPO's guarantee holds for an idealised update that nobody implements. Whether a practical algorithm can carry a meaningful improvement guarantee with modern function approximators, rather than a guarantee that evaporates under the approximations required to run it, is open.

Sources and Further Reading

Foundations

1. Williams, R. J. (1992). "Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning." Machine Learning, 8, 229–256. Springer
2. Sutton, R. S., McAllester, D., Singh, S., & Mansour, Y. (1999). "Policy Gradient Methods for Reinforcement Learning with Function Approximation." Advances in Neural Information Processing Systems 12 (NIPS 1999). (No stable open URL is linked here; the paper is indexed under this title and venue.)

Variance reduction and step control

3. Schulman, J., Levine, S., Moritz, P., Jordan, M. I., & Abbeel, P. (2015). "Trust Region Policy Optimization." ICML 2015. arXiv:1502.05477
4. Schulman, J., Moritz, P., Levine, S., Jordan, M. I., & Abbeel, P. (2015). "High-Dimensional Continuous Control Using Generalized Advantage Estimation." arXiv:1506.02438
5. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). "Proximal Policy Optimization Algorithms." arXiv:1707.06347

What actually makes it work

6. Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., & Madry, A. (2020). "Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO." ICLR 2020. arXiv:2005.12729

The LLM era

7. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., et al. (2024). "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models." arXiv:2402.03300

Related concepts on this site

8. Policy gradients and REINFORCE
9. Baselines and variance reduction
10. Actor-critic methods
11. Returns, discounting and episodes
12. The KL-regularised RL objective
13. Credit assignment over long generations

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.