Counterfactual Risk Minimisation

Evaluating one candidate policy from logs is a statistics problem. Searching over a million candidates for the best one is a learning problem, and it inherits a failure evaluation never sees: the optimiser finds the policy whose estimate is best, and with importance weighting, the policy with the best estimate is often the one that exploits the estimator's variance. Counterfactual risk minimisation (CRM) is the learning principle that accounts for this (Swaminathan and Joachims, 2015, Batch Learning from Logged Bandit Feedback through Counterfactual Risk Minimization, JMLR 16).

The setting is batch learning from bandit feedback: logs of \((x_i, y_i, \delta_i, p_i)\) where \(y_i\) was sampled by a logging policy \(h_0\) with propensity \(p_i = h_0(y_i \mid x_i)\) and \(\delta_i\) is the observed loss. There are no labels, only the loss of the action that happened to be taken.

From a bound to an objective

For a stochastic policy \(h\), the clipped IPS risk estimate is

Different policies get estimates of very different quality: a policy close to \(h_0\) has low-variance \(u_i^h\); a policy that favours rarely logged actions has high variance. Swaminathan and Joachims derive a generalisation bound using empirical Bernstein inequalities, in which the true risk is bounded by the estimate plus a term in the empirical variance of \(u^h\). Minimising the bound gives the CRM objective:

\(\lambda = 0\) recovers plain IPS learning. The variance term is a data-dependent regulariser that penalises policies whose estimated risk is unreliable, which is the same instinct as pessimism in offline reinforcement learning. Their POEM algorithm applies it to exponential-family (softmax) policies.

The paper's multi-label experiments, which convert supervised datasets to bandit feedback, give a sense of scale. On the TMC dataset, test Hamming loss is 3.445 for the logging policy, 2.808 for IPS learning, 2.197 for POEM, and 1.189 for a supervised CRF trained on full labels. The variance penalty closes more than a third of the gap between IPS learning and full supervision, and full supervision stays well ahead.

Loss scaling and propensity overfitting

The objective has a trap that supervised learning lacks. Losses must be scaled into \([-1, 0]\). If all losses are non-negative, a policy that puts zero probability on every logged action achieves \(\hat{R}^M = 0\) with zero variance, the best possible score, without learning anything about which actions are good.

This is propensity overfitting: the optimiser overfits to where the logging policy sampled, not to the losses. IPS risk is not equivariant, so translating all losses by a constant changes which policy minimises the estimate. Self-normalisation fixes equivariance, but its denominator couples all examples and breaks stochastic gradient descent.

BanditNet resolves this for deep networks (Joachims, Swaminathan and de Rijke, 2018, Deep Learning with Logged Bandit Feedback, ICLR). Fixing the normaliser \(S\) as a constraint and taking the Lagrangian turns the self-normalised objective into ordinary IPS with translated losses:

which is decomposable and trainable with SGD. A small grid of \(\lambda_j\) is swept, each solution's normaliser \(S_j\) is computed afterwards, and the SNIPS estimate picks the winner.

On CIFAR-10 with a ResNet20 and a logging policy at 49% error, BanditNet reached between 13% and 8.2% test error depending on how much bandit feedback it saw, approaching the fully supervised network. Plain IPS (\(\lambda = 0\)) did worse than the logging policy. Good \(\lambda\) fell between 0.8 and 1.0, and the REINFORCE-style choice of translating by the policy's expected loss, around 0.08 to 0.13, fell well outside it. The normaliser doubles as a diagnostic: its expectation is 1, the estimated standard error of every \(S_j\) was below 0.013, and a solution whose \(S_j\) falls outside roughly \([0.974, 1.026]\) should be discarded as a likely case of propensity overfitting.

When it breaks

The logging policy bounds what can be learned. CRM can only prefer actions the logger took with non-negligible probability. A deterministic logger gives no learning signal at all, and a nearly deterministic one gives weights so large that clipping dominates.

Hyperparameters are selected with the same biased estimator. \(M\), \(\lambda\) and the translation are chosen by counterfactual estimates on held-out logs, which share the weights' variance problem. Validation is less trustworthy than in supervised learning, and a final online test remains necessary.

The variance penalty is not always used. BanditNet's CIFAR experiments dropped it for lack of a validation set, and many production systems rely on clipping and self-normalisation alone. Whether the explicit penalty earns its tuning cost at deep-learning scale is not settled; recent theory favouring optimised additive baselines over self-normalisation (Jeunen and Gupta, 2026, arXiv:2602.14914) suggests the translation question is still moving.

Propensities must be logged, not reconstructed. Every weight divides by \(p_i\). A propensity recomputed later from a model that has since been retrained is a different number, and the bias it introduces is invisible in the objective.