Exploration in Language-Model RL

A policy trained with PPO on a maths reasoning task can plateau at 60% accuracy for thousands of gradient steps, then abruptly jump to 75% when a single new solution strategy first appears in a sampled batch. That jump is not a learning event; it is a discovery event. The learning happened in one forward pass once exploration finally produced the right kind of trajectory. This is the core tension in language-model RL: the gradient optimiser is powerful, but it is helpless unless the sampling distribution surfaces useful experience in the first place.

Why standard RL exploration intuitions do not transfer directly

In tabular or low-dimensional continuous RL, exploration means visiting states that have not been visited before. Epsilon-greedy, UCB, and intrinsic curiosity bonuses all operate on the assumption that you can enumerate or measure the novelty of a state.

A language model's "state" is the sequence of tokens generated so far, and its "action space" at each step is the full vocabulary (often 32k to 128k tokens). The combined space of possible completions for a 512-token response is astronomically large; it is never the case that any state is visited twice in any meaningful sense. The exploration problem is therefore not "visit new states" but "produce diverse, structurally varied completions."

The distinction matters practically. Two completions can be token-by-token different yet semantically identical. Measuring diversity at the token level is noisy. Measuring it at the semantic level requires embeddings or auxiliary models, adding computational cost and introducing new failure modes.

Temperature, entropy bonuses, and their limits

The cheapest exploration lever in language-model RL is sampling temperature. Higher temperature flattens the softmax, increasing per-token entropy and making the policy generate more varied outputs. This works early in training when the policy still has a broad distribution. It degrades as training progresses, because the policy concentrates probability mass on a small cluster of high-reward patterns and temperature can no longer surface genuinely different strategies without also surfacing incoherent text.

A more principled version is the entropy bonus, added directly to the reward signal:

r_total(y) = r_reward(y) + alpha * H(pi(. | x))

where H is the entropy of the policy's token-level distribution and alpha controls the exploration-exploitation tradeoff. This appears in maximum-entropy RL frameworks and is related to the temperature term in soft actor-critic. In practice, alpha is small (around 0.01 to 0.1) because large values destabilise the policy and reduce coherence faster than they improve diversity.

The KL penalty relative to the reference policy (the frozen SFT model) serves a related but different function. It prevents the policy from collapsing too far toward any single high-reward completion style. Written as part of the standard KL-regularised objective:

r_kl(y) = r_reward(y) - beta * KL(pi_theta(. | x) || pi_ref(. | x))

The KL term is an implicit diversity mechanism: it forces the policy to stay within a certain distributional radius of the reference model, which was itself trained to produce diverse text. But this is conservative diversity, not active exploration. The policy is constrained to not move too far away; it is not rewarded for moving toward novel regions.

How GRPO changes the exploration calculus

Group Relative Policy Optimisation (GRPO), introduced with DeepSeekMath, replaces the per-token value baseline with a group-level advantage. For each prompt x, you sample G completions {y1, ..., yG} from the current policy and compute rewards r1, ..., rG. The advantage of each completion is normalised relative to the group:

A_i = (r_i - mean(r_1..G)) / std(r_1..G)

This removes the need for a separate value network and centres the gradient signal on within-group variance. The exploration implication is immediate: if all G completions receive the same reward (all correct or all wrong), the advantages are all zero and the gradient update is zero. The policy learns nothing. In practice this means the group sampling diversity is a prerequisite for the training signal to exist at all.

Concretely: if your policy has already learnt to solve 90% of problems in a dataset and you sample G=8 completions, roughly 7 or 8 per prompt will be correct. Most batches will produce near-zero gradient, and the remaining training budget is wasted. This is sometimes called the "saturation problem." The practical fix is curriculum-style prompt difficulty control: keep the pass rate within a productive range (roughly 20-80% correct) so that within-group variance stays high.

Diversity-seeking strategies in practice

Several strategies have been used to maintain exploration pressure during language-model RL training.

Temperature annealing with restarts. Start with high temperature, gradually lower it as the policy improves, and periodically reset temperature upward if the entropy of sampled completions falls below a threshold. Simple to implement but requires careful tuning.

Best-of-N filtering for SFT warm-up. Before RL, run best-of-N sampling on hard prompts using the SFT model and add the successful completions to supervised fine-tuning data. This moves the starting policy toward harder problems before RL even begins, which expands the region of the completion space that RL can usefully explore.

Prompt curriculum. Dynamically adjust the difficulty of prompts in each batch based on recent pass rates. Prompts where the policy gets 0% or 100% correct are deprioritised; prompts in the 20-80% range are oversampled. The Kimi k1.5 technical report describes a variant of this approach combined with long-context RL scaling.

Process reward signals. Outcome rewards are sparse (correct or incorrect per full completion). Process reward models (PRMs) assign credit to intermediate reasoning steps, turning a sparse reward into a denser one. Denser rewards reduce the exploration burden because partially-correct solutions still receive a training signal. The tradeoff is that PRMs are expensive to train and can themselves be gamed.

Forced diversity via repeat-penalty or n-gram blocking. Borrowed from language generation, these heuristics penalise or forbid repetitive token sequences within a completion or across the G-sample group. They increase surface diversity cheaply but do not guarantee semantic diversity.

When it falls down

Reward hacking via superficially diverse completions. A policy can satisfy an entropy bonus or a diversity constraint while still converging on a small set of semantically equivalent strategies. If the reward signal rewards format compliance (e.g., answer wrapped in \boxed{}), the policy learns to produce many syntactically varied completions that are all structurally the same, gaining the diversity bonus without exploring new reasoning paths.

Temperature-coherence conflict. Above roughly temperature 1.5 to 2.0, language model completions become incoherent for most base models. The window in which temperature increases exploration without destroying coherence is narrow, and it narrows further as model size decreases. Small models have less redundancy in their representations and degrade faster under high-temperature sampling.

Group saturation at scale. As capability improves during a training run, an increasing fraction of prompts get solved by all G samples. The effective batch size of informative examples shrinks. Without dynamic prompt difficulty control, later training phases waste the majority of compute on zero-gradient updates.

KL collapse. If beta is too large relative to alpha, the KL penalty dominates and the policy barely moves from the reference model. This manifests as a flat reward curve despite apparently healthy training metrics. It is common when the reference model was already a strong supervised fine-tune.

Distributional shift in the replay buffer. PPO with a replay buffer or importance-weighted off-policy corrections can produce exploration from past, more diverse policies. But the off-policy distribution can differ enough from the current policy that importance weights become very large or very small, destabilising the gradient. The standard fix (clipping importance weights) reintroduces on-policy bias and partially defeats the exploration benefit.

Verifiable-reward brittleness. RLVR setups assume the verifier is reliable. If the verifier over-accepts (false positives on correct answers), the policy can exploit it without genuinely solving problems. Apparent exploration then leads to reward hacking rather than capability improvement.

Further reading

• Zhihong Shao et al., "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models" (2024) - introduces GRPO and demonstrates the group-saturation problem concretely: https://arxiv.org/abs/2402.03300
• Leo Gao, John Schulman, Jacob Hilton, "Scaling Laws for Reward Model Overoptimization" (2022) - the definitive empirical study of how over-optimising a proxy reward degrades true performance, with scaling curves: https://arxiv.org/abs/2210.10760
• Kimi Team, "Kimi k1.5: Scaling Reinforcement Learning with LLMs" (2025) - describes prompt curriculum and long-context RL scaling in a production setting: https://arxiv.org/abs/2501.12599
• Rafael Rafailov et al., "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (2023) - the derivation showing the KL-regularised RLHF objective has a closed-form solution, which illuminates why the KL term functions as an exploration constraint: https://arxiv.org/abs/2305.18290