Generalised Advantage Estimation

Policy gradient methods have a straightforward variance problem: the simplest unbiased advantage estimate requires rolling out entire trajectories, and the resulting signal is so noisy that learning often diverges before it converges. The standard fix, subtracting a value-function baseline, helps but does not eliminate the problem. Generalised Advantage Estimation (GAE), introduced by Schulman, Moritz, Levine, Jordan, and Abbeel in 2015, attacks the remaining variance systematically by exponentially down-weighting contributions from rewards further into the future, trading a small amount of bias for a large reduction in variance.

Every major modern policy-optimisation algorithm, including PPO, uses GAE. Understanding it is therefore not optional for anyone serious about reinforcement learning.

The advantage function and why estimating it is hard

The advantage of taking action \(a\) in state \(s\) under policy \(\pi\) is:

It measures how much better (or worse) action \(a\) is compared with the average action the policy would take. A perfect advantage signal would tell the optimiser exactly which decisions to reinforce. In practice, we never have the true \(Q^\pi\) or \(V^\pi\); we must estimate them from sampled rollouts.

Two extreme strategies exist:

Neither extreme is satisfactory on its own. Monte Carlo estimates are unbiased but too noisy for reliable gradient computation. TD estimates are smooth but inherit whatever errors live in the value function.

The GAE formula

GAE defines the advantage at time \(t\) as a geometrically-weighted sum of \(k\)-step TD residuals:

where \(\delta_{t+l} = r_{t+l} + \gamma V(s_{t+l+1}) - V(s_{t+l})\) is the TD residual at step \(t+l\).

The parameter \(\lambda \in [0, 1]\) is the central control knob:

• \(\lambda = 0\): collapses to the single-step TD residual \(\delta_t\). Maximum bias, minimum variance.
• \(\lambda = 1\): collapses to the full Monte Carlo advantage (minus the baseline). Zero bias (given a perfect \(V\)), maximum variance.
• \(\lambda \in (0, 1)\): a smooth interpolation. In practice, values of 0.95 to 0.99 work well across a wide range of continuous-control tasks.

The discount \(\gamma\) plays its usual role of reducing the effective horizon; \(\lambda\) is an additional, independent control over the bias-variance balance.

Recursive computation

The infinite sum looks expensive but collapses to a one-pass backward scan through the trajectory. Starting from the end of the collected rollout (length \(T\)):

delta[t] = r[t] + gamma * V[t+1] - V[t]
A[T-1] = delta[T-1]
for t in reversed(range(T-1)):
    A[t] = delta[t] + gamma * lambda * A[t+1]

This is \(O(T)\) and requires storing only the value estimates and rewards from the rollout, which are already in memory. The same backward pass appears verbatim in the Stable Baselines 3 PPO implementation.

Why the exponential weighting works

Intuitively, the TD residual \(\delta_{t+l}\) reflects the surprise at step \(t+l\): the difference between what the value function predicted and what actually happened. Residuals close to \(t\) contain mostly fresh information about the current action. Residuals far from \(t\) are contaminated by many subsequent random actions, each adding noise.

By weighting residuals with \((\gamma \lambda)^l\), GAE suppresses distant noise while retaining the nearby signal. The result is analogous to an exponential moving average applied to temporal credit signals, which is exactly why the paper draws a formal parallel to TD(\(\lambda\)) from Sutton's original work.

The connection to TD(\(\lambda\)) is precise: if you define \(G_t^{(k)}\) as the \(k\)-step return from time \(t\), then \(\hat{A}_t^{GAE(\gamma,\lambda)}\) equals the TD(\(\lambda\)) return minus the baseline \(V(s_t)\). The paper formalises this in Section 3 of Schulman et al. (2015).

GAE in practice: PPO

PPO (Schulman et al., 2017) is the canonical deployment of GAE. The algorithm collects a rollout of \(T\) steps, computes \(\hat{A}_t^{GAE}\) for every timestep using a learned critic \(V_\phi\), then optimises the clipped surrogate objective:

where \(r_t(\theta) = \pi_\theta(a_t|s_t) / \pi_{\theta_{old}}(a_t|s_t)\).

The quality of \(\hat{A}_t\) directly limits how well the clipping can protect against destructively large updates. A noisy advantage estimate means the direction of the gradient is unreliable even before the clipping comes into play. GAE, by controlling variance, is what makes PPO's optimisation landscape tractable.

The Stable Baselines 3 PPO defaults use gae_lambda=0.95 and gamma=0.99, which together give an effective exponential decay of \((0.99 \times 0.95)^l \approx 0.94^l\) per step into the future.

When it falls down

Bootstrapping error amplification. GAE inherits all errors from the value function \(V_\phi\). If the critic is poorly initialised or trained on too few updates per rollout, the TD residuals are systematically biased, and GAE propagates that bias across the entire trajectory via the backward scan. The advantage estimates can be confidently wrong. The solution is to train the critic more aggressively relative to the actor, but this increases compute per environment step.

Non-stationarity in the value function. GAE uses \(V_\phi\) snapshots taken during rollout collection. If the policy changes rapidly across multiple epochs of updates (as in PPO with many epochs), \(V_\phi\) from the previous rollout becomes stale as a baseline for the current advantages. Most implementations recompute advantages once per rollout and hold them fixed during the multi-epoch update, which is technically incorrect but works acceptably because PPO's clipping constrains policy movement.

Short-horizon rollouts hurt more at high \(\lambda\). At \(\lambda \approx 1\), the sum needs many future residuals to converge to the Monte Carlo estimate. If rollout length \(T\) is short relative to the task horizon, the sum is truncated and the estimate is biased low for rewards beyond the rollout boundary. Practitioners either lengthen rollouts or reduce \(\lambda\) when task horizons far exceed rollout length.

Sparse rewards. When most \(r_t = 0\), almost all TD residuals are nearly zero regardless of \(\lambda\). The advantage estimates carry almost no information about which actions led to the eventual reward. GAE does not solve the sparse-reward credit-assignment problem; it merely controls variance in the gradient signal, which is useless when the signal itself is nearly absent. Shaped rewards, hindsight relabelling, or curiosity-driven exploration are needed first.

Continuous-time limits. GAE's exponential weighting is derived for discrete time with fixed step size. In environments with variable timestep length (common in physics simulation), the effective \(\lambda\) depends on how finely the simulation is discretised. There is no principled way to pick \(\lambda\) that is step-size-invariant, which can make hyperparameter transfer across simulation resolutions unreliable.

Further reading

• Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation. https://arxiv.org/abs/1506.02438 (primary source; Section 3 for the full derivation).
• Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. https://arxiv.org/abs/1707.06347 (PPO, the most common deployment context for GAE).
• OpenAI Spinning Up PPO documentation: https://spinningup.openai.com/en/latest/algorithms/ppo.html (concise implementation notes including GAE lambda defaults).
• Stable Baselines 3 PPO API reference: https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html (production defaults: gae_lambda=0.95, gamma=0.99).