Evaluating an Offline RL Policy
Why the setting that forbids environment interaction also forbids the obvious way to compare policies, the estimators available, and why their variance is worst exactly where the decision matters.
Offline RL exists because interacting with the environment is expensive or dangerous. That same constraint means you cannot run a candidate policy to see how good it is, so selecting among candidates, and tuning the hyperparameters that control conservatism, has to be done from logged data alone. This is the field's least solved problem and the one most often quietly avoided.
The estimators
Importance sampling reweights logged returns by the ratio of the target policy's action probabilities to the behaviour policy's. It is unbiased and its variance grows exponentially with horizon, because the weight is a product of per-step ratios. On any trajectory of realistic length it is unusable without modification.
Per-decision and weighted importance sampling reduce the variance substantially by reweighting at each step rather than over whole trajectories, and by self-normalising, which introduces a small bias in exchange for a large variance reduction. This is nearly always the right trade.
Fitted Q evaluation trains a Q function for the target policy on the logged data by bootstrapping, then reads off the value at the initial states. It has low variance and inherits exactly the extrapolation problem that afflicts offline RL itself, so it is optimistic in the same direction and for the same reason.
Doubly robust estimators combine a model-based estimate with an importance-weighted correction, and are consistent if either the model or the propensities are correct. This is the standard recommendation and it does not eliminate the variance problem, it reduces it.
Model-based evaluation rolls the policy out in a learned dynamics model, which is cheap and inherits the model's compounding error, so it is optimistic wherever the policy leaves the data.
Why the variance is worst where it matters
Every importance-weighted estimator's variance grows with the divergence between the target and behaviour policies. A candidate policy nearly identical to the behaviour policy is estimated precisely and is not worth deploying. A candidate that differs substantially, which is the one that might be a real improvement, is estimated with variance that can exceed the effect being measured.
This is not a defect of any particular estimator; it reflects the information content of the data. The logged data contains little evidence about actions the behaviour policy rarely took, and no estimator manufactures evidence that is absent.
When it breaks
Published results frequently tune online. A large share of offline RL papers select hyperparameters using environment interaction, which is unavailable in the setting the methods exist for. Results obtained this way overstate what a practitioner can achieve, and reading the tuning protocol is necessary before believing a comparison.
The behaviour policy is usually unknown. Propensities must be estimated from the logs, and errors in that estimate propagate multiplicatively through the importance weights. A system that logged its own action probabilities at decision time avoids this entirely, which is a strong argument for propensity logging in any system that might later be optimised offline.
Effective sample size collapses silently. When a few trajectories carry most of the weight, the estimate is effectively based on those few regardless of dataset size. Reporting effective sample size alongside any importance-weighted estimate is what makes that visible.
Confidence intervals are wide and are usually omitted. Point estimates from off-policy evaluation invite comparisons the uncertainty does not support, and a ranking of candidate policies whose intervals all overlap is not a ranking.
12 flashcards for this concept
Click a card to reveal the answer.