Distribution Shift in Offline RL

Offline reinforcement learning learns a policy from a fixed dataset of logged interactions, with no further environment access. It is the setting that matters for anything where exploration is expensive or dangerous, healthcare, robotics on real hardware, industrial control, recommendation at scale, and it fails in a specific way that took the field years to characterise.

The failure

A value function is trained by bootstrapping: \(Q(s,a) \leftarrow r + \gamma \max_{a'} Q(s', a')\). The max is over all actions, including ones the dataset never contains at that state.

For those out-of-distribution actions the network is extrapolating, and extrapolation errors are not symmetric in their consequences. An underestimate is ignored, because the max discards it. An overestimate is selected, because the max seeks it. So the max operator systematically propagates the errors that are too high.

Worse, the inflated value becomes a bootstrap target for the previous state, which inflates that, and the error compounds backwards through the trajectory. A policy extracted from the resulting Q function confidently prefers actions whose apparent value is entirely an artefact of extrapolation, and it cannot be corrected because there is no environment to try them in.

This is why naively running an off-policy algorithm such as DQN or SAC on a fixed dataset frequently produces a policy far worse than the behaviour policy that generated the data, even when the data is good.

Why more data does not fix it

The problem is not statistical noise but the absence of coverage. If the dataset contains no examples of an action at a state, no amount of additional data about other actions constrains the estimate there. The issue is the support of the data distribution, not its size.

This distinguishes offline RL from supervised learning fundamentally. A supervised model evaluated on the training distribution is fine; the offline RL agent's whole purpose is to select actions the behaviour policy did not favour, so it necessarily queries outside the data.

The families of solution

Policy constraint. Keep the learned policy close to the behaviour policy, by an explicit divergence penalty, by only selecting actions the behaviour policy might have taken, or by weighting the update toward behaviour actions. Safe and limiting: it bounds improvement by how close you must stay.

Value pessimism. Penalise the value of out-of-distribution actions so the max no longer selects them. Conservative Q-learning does this by pushing down Q on unseen actions and up on dataset actions, yielding a lower bound on the true value (Kumar et al., 2020, arXiv:2006.04779). The policy then optimises a pessimistic estimate, which is exactly the right posture when you cannot verify.

Avoiding the max entirely. Implicit Q-learning fits an expectile of the value over dataset actions rather than a maximum over all actions, so no out-of-distribution query ever occurs. This is elegant and sidesteps the problem rather than mitigating it.

Sequence modelling. Decision Transformer reframes the problem as conditional sequence prediction, generating actions conditioned on a desired return, which is supervised learning and inherits none of the bootstrapping pathology, at the cost of not performing the stitching that dynamic programming provides.

When it breaks

Pessimism costs performance where data is good. A conservative algorithm on a dataset with broad coverage leaves improvement unclaimed, so the conservatism level is a hyperparameter that has to be tuned to the dataset and cannot be tuned by environment interaction.

Offline evaluation is the unsolved half. Selecting between candidate policies without environment access requires off-policy evaluation, which has high variance precisely when the policies differ most from the behaviour policy. Many published results tune hyperparameters using online evaluation, which is not available in the setting the method exists for.

Dataset composition dominates the algorithm. Results on expert-only, mixed and random datasets differ so much that method comparisons are only meaningful within a dataset type. A method that wins on expert data can lose badly on mixed data.

The behaviour policy is usually unknown. Constraint methods need it, so it is estimated from the data, and the estimate's errors feed directly into the constraint.