Conservative Value Estimation
How pessimism turns an unverifiable estimate into a safe one, the mechanism behind conservative Q-learning, and why a lower bound is the right object when you cannot test.
When you can test a hypothesis, optimism is a good strategy: try the promising thing and find out. When you cannot test, optimism is a strategy for being confidently wrong. Offline RL is the second case, and pessimism is the corresponding principle.
The principle
Rather than estimating the value of a policy, estimate a lower bound on it, and optimise that. A policy that is good under a pessimistic estimate is good under the true one, because the estimate never exceeds the truth. A policy that looks good under an optimistic estimate may be arbitrarily bad.
This inverts the optimism principle from bandits deliberately, and the inversion is driven entirely by whether the agent can act to resolve its uncertainty. It can in a bandit and cannot offline.
Conservative Q-learning
CQL adds a regulariser to the standard Bellman objective that pushes down the Q values of actions sampled from the learned policy and pushes up the Q values of actions actually present in the dataset:
The effect is that out-of-distribution actions cannot achieve high values, so the max in the bootstrap target no longer selects them, and the compounding overestimation is cut at its source. Under conditions the paper states, the resulting Q function lower-bounds the true value function.
The coefficient \(\alpha\) is the conservatism dial. Too small and overestimation returns; too large and the value function is so pessimistic that the extracted policy is indistinguishable from the behaviour policy. There is no way to tune it offline without an evaluation method, which is the field's recurring difficulty.
Uncertainty-based alternatives
An ensemble of Q functions gives a disagreement signal, and penalising the value by the ensemble's standard deviation implements pessimism proportional to epistemic uncertainty rather than uniformly. This targets the penalty where it is needed and costs training several networks, and the ensemble's disagreement is itself an approximation whose quality varies.
When it breaks
Uniform pessimism penalises good unexplored actions equally with bad ones. The bound is safe and loose, and a genuinely better action that the behaviour policy rarely took is suppressed alongside the dangerous ones. This is the cost of not being able to distinguish them.
Lower bounds are not tight. A method that provably lower-bounds the value has said nothing about how far below the truth it sits. Two conservative methods can both be valid lower bounds and differ enormously in usefulness.
Pessimism interacts with function approximation. The theoretical guarantees generally assume conditions that neural networks do not satisfy, so the lower-bound property is a motivation for the method rather than a guarantee about the implementation.
The distribution shift returns at deployment. A policy trained conservatively on logged data still encounters states at deployment that the dataset did not cover, and pessimism about actions does not address unfamiliar states. This is the gap between offline training and online performance that conservatism alone does not close.
12 flashcards for this concept
Click a card to reveal the answer.