Proximal Policy Optimisation

Before PPO, the practical choice for continuous-control RL was brutal: either accept the instability of vanilla policy gradients, or pay the cost of Trust Region Policy Optimisation (TRPO), which required conjugate gradients and a constrained optimisation step so expensive it was effectively inaccessible for large neural networks. Schulman et al. shipped PPO in 2017 as a direct response to that tradeoff. It is now the default training algorithm behind ChatGPT's RLHF stage, most of Google DeepMind's locomotion work, and a large fraction of everything labelled "fine-tuned with RL" in the past five years.

The Core Tension: Stability vs. Simplicity

Vanilla policy gradient methods update parameters by following the gradient of the expected return:

∇J(θ) = E[ ∇ log π_θ(a|s) · A(s, a) ]

where A(s, a) is the advantage - how much better action a is relative to the baseline. The problem is step size. A large learning rate can catapult the policy into a region where the old samples no longer represent the distribution well, and since the old samples are exactly what you are training on, the gradient signal becomes wrong in a self-reinforcing way. Recovery is slow or impossible.

TRPO solved this by enforcing an explicit KL constraint:

maximise  E[ (π_θ / π_θ_old) · A ]
subject to  KL( π_θ_old || π_θ ) ≤ δ

Correct, but the constrained step involves computing the Fisher information matrix and running an inner conjugate-gradient loop, which is incompatible with most deep learning toolchains and requires careful implementation.

PPO's insight: you do not need the exact constrained solution. You just need to prevent the ratio from straying too far. Clipping does that cheaply.

The Clipped Surrogate Objective

Define the probability ratio:

r_t(θ) = π_θ(a_t | s_t) / π_θ_old(a_t | s_t)

The unclipped surrogate objective is E[ r_t(θ) · A_t ]. PPO-Clip modifies it:

L_CLIP(θ) = E[ min( r_t(θ) · A_t,  clip(r_t(θ), 1-ε, 1+ε) · A_t ) ]

The min is the operative word. Consider two cases:

The result is a pessimistic lower bound on the unclipped objective. Gradient updates can only improve performance within the clip boundary; any attempted step outside it yields zero gradient from the clipped term. Typical ε is 0.1 to 0.2.

There is also a KL-penalty variant (PPO-KL), which adds a penalty β · KL(π_θ_old, π_θ) to the loss and adaptively adjusts β depending on whether the measured KL divergence exceeds a target. In practice, PPO-Clip is more common because it does not require tuning the penalty coefficient.

The Full Training Loop

PPO is an on-policy algorithm but it amortises data collection by running multiple gradient steps on the same batch of trajectories, which partially mimics the efficiency of off-policy methods:

for iteration 1, 2, ...:
    collect T timesteps of data using current π_θ_old
    compute advantages A_t using GAE (see below)
    for epoch 1..K:
        for each minibatch of size M:
            compute L_CLIP + c1·L_VF - c2·S   # S = entropy bonus
            update θ via Adam
    θ_old ← θ

Three terms in the loss:

• L_CLIP: the clipped policy objective (maximised).
• L_VF: value function mean-squared error, (V_θ(s_t) - V_t^target)^2, shared-network case (minimised, coefficient c1 ~ 0.5).
• S: an entropy bonus on the policy to discourage premature collapse to deterministic actions (coefficient c2 ~ 0.01).

Generalised Advantage Estimation (GAE). Computing A_t directly as R_t - V(s_t) is high variance. PPO almost always uses GAE (Schulman et al. 2016):

A_t^GAE(λ) = Σ_{k=0}^{∞} (γλ)^k · δ_{t+k}
δ_t = r_t + γ V(s_{t+1}) - V(s_t)

λ = 1 recovers Monte Carlo returns (low bias, high variance); λ = 0 recovers one-step TD (low variance, higher bias). λ ~ 0.95 is standard. GAE is what makes the value network architecturally load-bearing: without a good V estimate, the advantages are noisy and learning stalls.

Why PPO Became the Default

Three practical properties separate PPO from its predecessors:

1. First-order only. All updates are standard gradient ascent with Adam. No conjugate gradients, no line search, no Fisher information. PPO drops into any PyTorch or JAX training loop without modification.

2. Multiple epochs on one batch. TRPO and vanilla policy gradients can only safely do one gradient step per data batch. PPO's clipping allows K epochs (typically 3-10), making it several times more sample efficient per wall-clock hour.

3. Robust to reward scale. Advantage normalisation within the minibatch (subtract mean, divide by std) is a standard companion trick that makes PPO tolerant of reward functions with very different magnitudes across tasks.

The RLHF pipelines that produced InstructGPT and subsequently ChatGPT used PPO to optimise language models against a reward model trained on human preferences. The policy is a language model; the action space is the vocabulary; the "environment" is the reward model. The clipping constraint is critical there because a single large update to a language model can irreversibly destroy fluency.

When It Falls Down

On-policy data hunger. PPO discards trajectories after K epochs. Problems requiring millions of environment steps (most Atari hard-exploration games, sparse-reward 3D navigation) hit wall-clock limits before solving. Off-policy methods like SAC are preferred when a replay buffer is feasible.

Reward hacking under clipping. The clipping protects the update magnitude, not the objective. If the reward function is misspecified, PPO will faithfully optimise the wrong signal within the trust region. In RLHF this manifests as reward model overoptimisation: the policy learns to score high on the reward model while drifting away from genuinely helpful behaviour, a failure the KL penalty against the reference policy partially mitigates.

Sensitive to advantage normalisation and value initialisation. Without careful normalisation, the clipping operates in the wrong regime. A badly initialised value head can produce advantage estimates with the wrong sign for thousands of steps, during which the policy degrades rather than improving.

Discrete vs. continuous action spaces. PPO works for both, but the optimal ε and number of epochs differ substantially. A setting tuned for a robotics task will often fail on an Atari game without retuning.

Entropy collapse in large action spaces. In LLM fine-tuning the vocabulary is roughly 50k tokens. Without an appropriately scaled entropy coefficient the policy concentrates probability mass rapidly, resulting in repetitive outputs. This is qualitatively different from the small discrete action spaces PPO was benchmarked on originally.

Further Reading

• Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). "Proximal Policy Optimization Algorithms." arXiv:1707.06347. The original paper; Section 3 gives the cleanest derivation of the clipped objective. https://arxiv.org/abs/1707.06347

• Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2016). "High-Dimensional Continuous Control Using Generalized Advantage Estimation." arXiv:1506.02438. Essential companion: understanding GAE is required to understand why PPO's value network matters. https://arxiv.org/abs/1506.02438

• Schulman, J., Levine, S., Moritz, P., Jordan, M., and Abbeel, P. (2015). "Trust Region Policy Optimization." ICML 2015. arXiv:1502.05477. Read this first to appreciate the problem PPO is solving. https://arxiv.org/abs/1502.05477

• OpenAI Spinning Up - PPO documentation. Annotated pseudocode with hyperparameter guidance and working PyTorch/TensorFlow implementations. https://spinningup.openai.com/en/latest/algorithms/ppo.html