Learning From the Log: Off-Policy Policy Learning, From IPS to Counterfactual Risk Minimisation

In a live experiment on YouTube's candidate generator, one number changed. The cap on importance weights used to train the recommender went from \(e^3\), about 20, to \(e^5\), about 148. Same model, same REINFORCE objective, same traffic. View time fell by 0.52%, a significant loss on a system where a 0.07% gain was worth reporting (Chen et al., 2019, Top-K Off-Policy Correction for a REINFORCE Recommender System, WSDM, arXiv:1812.02353). The authors' explanation: with the cap lifted, the policy "could potentially overfit to a few logged actions that accidentally receives high reward."

Every recommender and ad server writes this kind of log, and it records only the outcome of the action taken. Learning a better policy from it means reasoning about actions rarely taken, and the tool that permits this, importance weighting, is exactly what an optimiser exploits.

Why this matters: Off-policy evaluation asks whether one candidate policy is good. Off-policy learning searches millions of candidates, and the search manufactures bias out of an unbiased estimator. Knowing which repair fixes which failure separates a learner that ships from one that learns the logger's sampling noise.

TL;DR

• IPS estimates any single policy's value without bias, but the maximum of IPS estimates is biased upward, and the bias concentrates on policies that differ most from the logger.
• A deterministic policy that picks an action the logger chose 5% of the time carries weight 20 on every matching event; its effective sample size is about 5% of the log.
• Counterfactual risk minimisation (CRM) penalises each policy's own estimated standard error. On the TMC benchmark it cut test Hamming loss from 2.808 (IPS learning) to 2.197, against 1.189 with full supervision (Swaminathan and Joachims, 2015, JMLR 16).
• Propensity overfitting is a different failure: with non-negative losses, a policy scores well by avoiding logged actions. BanditNet's loss translation fixed it and trained a ResNet on CIFAR-10 from a 49%-error logger to 13% down to 8.2% error (Joachims, Swaminathan and de Rijke, ICLR 2018).
• Doubly robust policy learning has regret on the order of \(\sqrt{\mathrm{VC}(\Pi)/n}\) even with estimated propensities (Athey and Wager, 2021, Econometrica 89(1)), but on click rates of a few percent the reward model removes little variance.
• No estimator dominates. On the 26-million-row Open Bandit Dataset the direct method was most accurate at \(n = 10{,}000\) in two of three campaigns and a shrinkage doubly robust estimator at \(n = 300{,}000\) (Saito et al., NeurIPS 2021).

At a Glance

flowchart LR
    L["Logging policy pi0"] --> G["Log with x, a, p, r"]
    G --> E["Counterfactual estimator"]
    E --> O["Optimiser over policies"]
    O --> P["Learned policy"]
    O -.->|"argmax finds noise"| N["Estimator variance"]
    N -.-> R["Penalty, SNIPS, cap, DR"]
    R --> E
    P --> V["Online A/B test"]
    V --> L

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff

    class L,G blue
    class E,O purple
    class P,V teal
    class N rose
    class R emerald

The dashed path decides whether this works: the optimiser finds the policy most flattered by chance, and every technique below feeds that back into the estimator.

Before Counterfactual Risk

The mathematics predates recommenders. Horvitz and Thompson estimated population totals from unequal-probability samples by dividing each observation by its inclusion probability (Horvitz and Thompson, 1952, JASA 47(260)). Replace "unit sampled into a survey" with "action shown to a user" and you have IPS.

The machine learning version came with contextual bandits. Beygelzimer and Langford's offset tree reduced learning from partial labels to binary classification (Beygelzimer and Langford, 2009, KDD, arXiv:0812.4044). Strehl and colleagues handled logs whose propensities were never recorded and had to be estimated (Strehl et al., 2010, NeurIPS). Dudík, Langford and Li combined a reward model with propensities so that either being accurate suffices (Dudík, Langford and Li, 2011, ICML, arXiv:1103.4601). Bottou and colleagues ran the reasoning on Bing's ad placement engine, randomising reserve prices and clipping weights, and were explicit that clipping ignores what the logger explored poorly (Bottou et al., 2013, JMLR 14).

In 2015 Swaminathan and Joachims made the search over policies the central problem.

timeline
    title Learning policies from logged bandit feedback
    1952 : Horvitz and Thompson weight by inverse inclusion probability
    2009 : Offset tree reduces partial labels to binary classification
    2011 : Doubly robust evaluation and learning
    2013 : Counterfactual reasoning on Bing ad placement
    2015 : Counterfactual risk minimisation and POEM
         : Self-normalised estimator names propensity overfitting
    2018 : BanditNet trains deep networks from logs
         : Empirical welfare maximisation in Econometrica
    2019 : YouTube ships top-K off-policy REINFORCE
    2021 : Doubly robust policy learning regret bounds
         : Open Bandit Dataset with two logging policies

[IMAGE: One log row as a record card: hashed context, shown item, propensity 0.05 in amber, click reward 1. A faded ghost column holds the two unshown items with question marks. Caption: "Logged bandit feedback. Everything a policy learner does is an argument about the ghost column."]

How Off-Policy Policy Learning Actually Works

The estimand and why IPS is unbiased

Each log entry is \((x_i, a_i, p_i, r_i)\): context \(x_i\), action \(a_i \sim \pi_0(\cdot \mid x_i)\), propensity \(p_i = \pi_0(a_i \mid x_i)\), reward with mean \(q(x_i, a_i)\). A policy's value is \(V(\pi) = \mathbb{E}_x[\sum_a \pi(a \mid x)\, q(x, a)]\), and the goal is \(\arg\max_{\pi \in \Pi} V(\pi)\) over some class: softmax models, trees, deep networks. IPS weights each reward by how much more often the target would have taken the logged action:

For a fixed context the logger's probabilities cancel:

The cancellation needs common support: wherever \(\pi(a \mid x) > 0\), \(\pi_0(a \mid x) > 0\). An action the logger never shows drops out of the sum silently.

Why IPS is noisy, and why the argmax makes it worse

Per event, \(\mathrm{Var}(w r) = \mathbb{E}_{\pi_0}[w^2 r^2] - V^2\). With rewards in \([0,1]\) the first term is at most \(\mathbb{E}_{\pi_0}[w^2] = 1 + \chi^2(\pi \,\|\, \pi_0)\), the chi-square divergence. A deterministic target choosing \(a^\star\) has \(\mathbb{E}[w^2] = 1/\pi_0(a^\star \mid x)\), and its effective sample size \(n_{\text{eff}} = (\sum w_i)^2/\sum w_i^2\) is just the number of matching events.

For evaluation a wide interval is an honest answer; for learning it is an opportunity. If \(\hat\pi = \arg\max_\pi \hat{V}(\pi)\), then by Jensen's inequality, since the maximum is convex,

The selected estimate is optimistic, and unevenly so: high-variance estimates have more room to be lucky, so the argmax drifts toward rarely logged actions that happened to be rewarded. With known propensities, IPS-based welfare maximisation still has regret on the order of \(\sqrt{\mathrm{VC}(\Pi)/n}\) (Kitagawa and Tetenov, 2018, Econometrica 86(2)). Such bounds treat all policies alike.

Counterfactual risk minimisation

Swaminathan and Joachims use losses \(\delta_i\) and clipped weights: \(u_i^h = \delta_i \min\{M,\, h(a_i \mid x_i)/p_i\}\), with mean \(\hat{R}^M(h)\) and sample variance \(\widehat{\mathrm{Var}}_h(u)\). Their Theorem 1, built on empirical Bernstein bounds, says that with probability at least \(1 - \gamma\), for every \(h\) in the class simultaneously,

where \(Q_{\mathcal{H}}\) is a log covering-number term (Swaminathan and Joachims, 2015, JMLR 16, 1731-1755). The middle term varies by hypothesis. Folding the constants into \(\lambda\) gives the CRM objective:

Minimising an upper confidence bound on risk is maximising a lower confidence bound on value, which makes CRM an early statement of pessimism in offline decision-making. POEM applies it to softmax policies \(h_w(a \mid x) \propto \exp(w \cdot \phi(x, a))\); because the square root couples all examples, it iterates a Taylor-based majorisation of the variance term. The paper also names the ceiling: with a deterministic logger or deficient support, learning can stay sub-optimal however large the log.

Propensity overfitting and self-normalisation

A second failure appeared the same year (Swaminathan and Joachims, 2015, The Self-Normalized Estimator for Counterfactual Learning, NeurIPS). Translate every loss by \(c\):

\(S(h)\) has expectation exactly 1, but its realised value depends on \(h\). With non-negative losses a policy lowers its risk by shrinking \(S\), which it does by moving probability away from whatever the logger chose. When every context is unique, as with real users, a flexible model can zero out the logged action at every unrewarded event. It has learned where \(\pi_0\) sampled, not what users want. The self-normalised estimator divides by \(\sum_i w_i\) instead of \(n\), so a translation by \(c\) moves it by exactly \(c\). This equivariance removes the incentive, at the price of \(O(1/n)\) bias and a denominator that couples all examples.

BanditNet: self-normalisation with SGD

Joachims, Swaminathan and de Rijke treat \(S\) as a constant to search over (ICLR 2018). Constrain \(S(w) = S_j\) and minimise the IPS numerator; the Lagrangian is

For fixed \(\lambda\) this is plain IPS with translated losses, decomposable and trainable as the output layer of any network. Training sweeps a grid of \(\lambda_j\), computes each solution's \(S_j\), and keeps the best self-normalised estimate. On CIFAR-10, pure IPS (\(\lambda = 0\)) did worse than the 49%-error logger. Good \(\lambda\) sat between 0.8 and 1.0; the REINFORCE habit of using the policy's own expected loss, 0.083 to 0.130, fell well outside. \(S_j\) doubled as a diagnostic: its estimated standard error was below 0.013, so a solution with \(S_j\) outside roughly \([0.974, 1.026]\) should be discarded. The CIFAR runs also dropped the variance penalty, lacking a validation set.

Doubly robust policy learning

IPS ignores what is known about rewards; the direct method (fit \(\hat{q}\), act greedily) ignores propensities. Doubly robust scores use both:

and the learner maximises \(\frac{1}{n}\sum_i \sum_a \pi(a \mid x_i)\,\hat{\Gamma}_i(a)\). For binary treatment, Athey and Wager choose \(\hat\pi = \arg\max_{\pi} \frac{1}{n}\sum_i (2\pi(X_i) - 1)\hat{\Gamma}_i\) with cross-fitted augmented IPW scores and prove regret on the order of \(\sqrt{\mathrm{VC}(\Pi)/n}\), with constants set by the semiparametrically efficient variance of evaluating the best policy (Athey and Wager, 2021, Econometrica 89(1), 133-161). The advance over Kitagawa and Tetenov is that the rate survives estimated propensities, because the score's error is a product of two nuisance errors and is small enough to withstand a supremum over \(\Pi\). Zhou, Athey and Wager extended this to many actions with tree policies (Zhou, Athey and Wager, 2023, Operations Research 71(1)).

Gradients at production scale

Large recommenders take importance-weighted policy gradients, \(\frac{1}{n}\sum_i \frac{\pi_\theta(a_i \mid x_i)}{\beta(a_i \mid x_i)} r_i \nabla_\theta \log \pi_\theta(a_i \mid x_i)\). YouTube's data came from several historical policies, so \(\beta\) was learned by a second head, reintroducing estimated propensities. Because a page shows \(K\) items, Chen et al. optimised the probability of appearing in a sampled top-\(K\) set, \(\alpha_\theta = 1 - (1 - \pi_\theta)^K\), which multiplies the usual weight by

A low-probability item gets up to \(K\) times the push; once it is likely to appear, the multiplier falls toward zero and frees mass for other items.

Seeing It in Motion

Two loops run at different speeds: training, and deployment, where the learned policy becomes the next logger.

sequenceDiagram
    participant U as User traffic
    participant S as Serving policy
    participant G as Log store
    participant T as Trainer
    participant C as Counterfactual gate
    U->>S: Context x
    S->>U: Action a sampled with p
    U->>G: Reward r with x, a, p
    G->>T: Batch of logged tuples
    Note over T: Sweep lambda, fit translated IPS
    T->>C: Candidates with S_j and SNIPS value
    C->>C: Drop S_j outside 0.974 to 1.026
    C->>S: Best candidate to A/B slice
    Note over S,G: New policy becomes next logger

The propensity is therefore a data contract: computed at serving time from the distribution that sampled the action, and written with the event.

flowchart TB
    subgraph Serve["Serving path"]
        X["Request context"] --> SM["Policy scores"]
        SM --> SA["Sample action"]
        SA --> PL["Write propensity with event"]
    end
    subgraph Learn["Learning path"]
        PL --> J["Join delayed rewards"]
        J --> W["Weights capped at M"]
        J --> RM["Reward model"]
        RM --> DR["DR scores"]
        W --> OB["CRM or translated IPS loss"]
        DR --> OB
        OB --> CK["S check and OPE report"]
    end
    CK -->|"passes"| AB["Online test"]
    CK -.->|"fails"| RJ["Reject candidate"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class X,SM,SA blue
    class PL,W amber
    class J,RM,DR,OB purple
    class CK slate
    class AB teal
    class RJ rose

[IMAGE: Histograms of 2,000 bootstrap IPS estimates for three deterministic policies from 1,000-event logs with logger (0.70, 0.25, 0.05). The policy on the 5% action has the widest, right-skewed histogram; true values marked as vertical lines; the distribution of the maximum of the three shaded rose. Caption: "Each estimate is unbiased; the maximum of three is not."]

By the Numbers

Sources: POEM Table 3, batch variant (Swaminathan and Joachims, 2015, arXiv:1502.02362); BanditNet Section 4 (Joachims et al., 2018); YouTube Section 6.2 (Chen et al., 2019); Decision Service figures are the builders' own deployment claims (Agarwal et al., 2016, arXiv:1606.03966); Open Bandit Dataset Tables 1, 2, 4 and 11 (Saito et al., 2021); Criteo size as listed in that comparison for the test-bed of Lefortier et al., 2016, arXiv:1612.00367.

Three patterns stand out. The gap to full supervision stays large, because that information was never in the log. Production gains are well under 1% and hinge on what theory calls hyperparameters: the cap and \(K\). And a better reward model cut DR's error by 78.5%, so doubly robust is only as good as its nuisance models.

A Concrete Example

One context type, three actions, 1,000 logged impressions.

Step 1. World and logger. True click rates are \(q = (0.06, 0.09, 0.07)\) for A, B, C, so B is best. The logger is \(\pi_0 = (0.70, 0.25, 0.05)\), worth \(0.042 + 0.0225 + 0.0035 = 0.068\).

Step 2. The log. Sampling gave 712 impressions of A, 243 of B, 45 of C. Clicks: 43, 22 and 6. C is lucky: 45 impressions at 0.07 should give about 3.2 clicks.

Step 3. IPS for the three deterministic policies.

IPS learning picks C. Its estimate is 71% above its true value, and deploying it earns 0.07 instead of 0.09, a regret of 22% of the best click rate. Every estimate was unbiased; the argmax was not.

Step 4. Standard errors. For C, \(u = 20\) on 6 events, so \(\mathbb{E}[u^2] = 400 \times 0.006 = 2.4\), variance \(2.4 - 0.12^2 = 2.3856\), standard error \(\sqrt{2.3856/1000} = 0.0488\). Likewise B: \(16 \times 0.022 - 0.088^2 = 0.3443\), SE 0.0186. A: \(2.0408 \times 0.043 - 0.0614^2 = 0.0840\), SE 0.0092.

Step 5. CRM with \(\lambda = 2\). Maximise \(\hat{V} - 2\,\text{SE}\): A \(0.0614 - 0.0183 = 0.0431\); B \(0.0880 - 0.0371 = 0.0509\); C \(0.1200 - 0.0976 = 0.0223\). CRM picks B. The ranking flips at \(\lambda = 0.032/0.0302 \approx 1.06\): below it C's luck wins, above it B's effective sample size, more than five times C's, does.

Step 6. Self-normalisation does not help here. \(\hat{V}/S\) gives A 0.0604, B 0.0905, C 0.1333. C was under-sampled (45 against an expected 50), so normalising pushes its estimate up. SNIPS fixes a different problem.

Step 7. Propensity overfitting. Use losses \(\delta = 1 - r\), so \(\hat{R} = S - \hat{V}\): A 0.956, B 0.884, C 0.780. C's lead over A is 0.176, and 0.117 of it comes from \(S\), from C being logged less than expected, not from clicks. With unique contexts, a flexible policy could also zero out C at its 39 unclicked contexts. Translating by \(\lambda = 0.93\) gives \(\hat{R} - 0.93\,S = 0.07\,S - \hat{V}\): A \(+0.010\), B \(-0.020\), C \(-0.057\). The coefficient on \(S\) fell from 1 to 0.07, so avoiding the log no longer pays; what remains is the noisy click signal, which Step 5 handles.

Step 8. Doubly robust. Take an imperfect \(\hat{q} = (0.058, 0.085, 0.080)\). For always-C, \(\hat{V}_{\text{DR}} = 0.080 + 20 \times (6 - 45 \times 0.08)/1000 = 0.128\). Per-event scores are 18.48 (clicked C), -1.52 (unclicked C) and 0.08 (all others), giving a standard error of about 0.046 against IPS's 0.049. At a 7% click rate nearly all variance is whether a click happened, which no model of the mean removes; DR helps far more on dwell time or revenue.

[IMAGE: Interval plot for A, B, C: dot at the IPS estimate, bar spanning two standard errors either side, diamond at the true value. C's bar is more than twice as wide as B's and its dot sits far right of its diamond; B's left end is highest. Caption: "IPS ranks by the dots; CRM with lambda 2 ranks by the left ends of the bars."]

Where It Breaks

[IMAGE: Scatter of per-context propensity for the logged action (x-axis, log scale 0.001 to 1) against the learned policy's probability for it (y-axis), for a translated and an untranslated model. The untranslated model's unclicked points collapse onto y = 0. Caption: "Propensity overfitting made visible: the model learns to avoid the log."]

The logger bounds what can be learned

Where \(\pi_0(a \mid x) = 0\) nothing is estimable; where it is tiny, weights explode and clipping takes over. The failure is quiet, because SNIPS renormalises over the actions it has seen. Many production loggers are deterministic rankers, which is why YouTube first had to serve a stochastic policy.

Estimated propensities import a second model's errors

When propensities are learned, as with YouTube's \(\beta\) head, an under-estimated propensity on a rare action inflates its weight and hands the optimiser a free lunch invisible in the objective. Athey and Wager's guarantee tolerates estimated propensities only through DR's second-order error and rate conditions on both nuisance models.

Hyperparameters are chosen by the estimator under suspicion

\(M\), \(\lambda\) and the translation are tuned with counterfactual estimates that share the weights' variance. Open Bandit Pipeline experiments show the stakes even for evaluation: DRos's shrinkage parameter changed RMSE by more than threefold across its grid, automatic tuning found the best value in one logging direction but not the other, and the best estimator changed between \(n = 10{,}000\) and \(n = 300{,}000\).

Overparameterised models favour a different objective

Brandfonbrener, Whitney, Ranganath and Bruna argue that with large networks, value-based methods inherit the benign generalisation of overparameterised regression while policy-based IPS objectives do not. The reason is action-stability: the value objective's optimal prediction does not depend on which action was logged, the policy objective's does, so an interpolating model learns the logging pattern. They prove regret upper bounds for the former and lower bounds for the latter (Brandfonbrener et al., 2021, ICML, arXiv:2006.15368). That sits in tension with BanditNet's policy-objective ResNet approaching supervised accuracy; the translation and \(S\) check defend against exactly this failure, but the two results point toward opposite defaults.

Feedback loops, proxies and slates

Every deployed learner becomes the next logger. YouTube notes that each experiment's system supplied the next one's training data, so results cannot be compared across stages. The reward is a proxy, and IPS is unbiased only for the proxy: the standard correction raised videos viewed 0.53% without moving view time, while the top-K version raised view time 0.85% as videos viewed fell 0.16%. Pages are slates, and both the top-K multiplier and the Open Bandit Dataset assume items are judged independently of their neighbours, an assumption the dataset's authors flag as likely false.

[IMAGE: Simulated curves against weight cap M on a log axis from 1 to 1,000: bias (rose) rising as M shrinks, variance (amber) rising as M grows, and the true value of the policy learned at each cap (teal) peaking in between, with markers at e^3 and e^5. Caption: "The cap is a bias-variance dial for the learned policy, not just the estimate."]

Alternative Designs

The Bayesian row is London and Sandler's PAC-Bayesian regulariser (London and Sandler, 2019, ICML, arXiv:1806.11500). Most production teams pair a direct model with capped or doubly robust weighting and let an online test decide.

How It Is Used in Practice

YouTube's candidate generator is the most detailed public account: REINFORCE over millions of items, a learned behaviour head, the top-K multiplier, and a cap at \(e^3\). Weight capping "brings the biggest gain online", and neither normalised importance sampling nor a TRPO-style KL penalty added more. And the correction changed what was recommended before it moved the headline metric: the share of nominations from outside the control model's top ranks rose nearly threefold.

Microsoft's Decision Service made explore, log, learn and deploy a piece of infrastructure so that propensities are logged correctly by construction; its builders report 25% to 30% click-through gains on content recommendation and an 18% revenue lift on a landing page (Agarwal et al., 2016). These are the builders' claims.

ZOZOTOWN ran uniform random and Bernoulli Thompson sampling side by side for seven days, and that random slice is what makes the Open Bandit Dataset useful: estimators can be checked against on-policy ground truth. The Criteo test-bed (Lefortier et al., 2016) serves the same role for display ads. In economics the doubly robust branch ships as the policytree package, which learns shallow tree policies.

[IMAGE: Swimlane of one quarter: add a 5% stochastic slice, verify logged propensities by replaying scores, run offline sweeps with the S check, A/B test, then the new policy becomes the logger and the baseline resets. Caption: "Each shipped policy changes the data the next one learns from."]

Insights Worth Remembering

1. Unbiased estimates do not give unbiased decisions. The maximum of unbiased estimates is biased upward, and the bias lands on the noisiest policies. Off-policy learning is evaluation plus a search, and the search is the hard part.

2. Variance and propensity overfitting need different cures. A variance penalty stops trust in lucky rare actions; self-normalisation or translation stops scoring by avoiding the log. A pipeline can have one fix and suffer the other.

3. The normaliser \(S\) is a free alarm. Its expectation is exactly 1. Checking that each candidate's \(S\) sits within a couple of standard errors of 1 catches a failure no loss curve shows.

4. The loss offset is a modelling decision. Recoding clicks from \(\{0,1\}\) to \(\{-1,0\}\) changes which policy minimises IPS risk. BanditNet's good offsets were near the typical loss, far from the REINFORCE baseline.

5. Doubly robust is not automatically low variance. It removes the variance a reward model explains. On clicks at a few percent that is very little.

6. No estimator creates support. A small stochastic slice with correctly logged propensities is worth more than any reweighting of a deterministic log.

Open Questions

Value-based or policy-based by default? Brandfonbrener et al. prove a gap favouring value-based objectives for overparameterised models; BanditNet shows a self-normalised policy objective near supervised accuracy. How they compare on real logs with estimated propensities and drifting rewards, where neither set of assumptions holds, is unmeasured.

Is self-normalisation the right control variate for learning? Jeunen and Gupta prove that IPS with an optimal additive baseline asymptotically dominates SNIPS in mean squared error for evaluation (Jeunen and Gupta, 2026, SIGIR, arXiv:2602.14914). BanditNet's translation is itself an additive baseline picked by a self-normalised criterion. Whether the evaluation result carries over to learning, where the baseline also shapes gradients and the overfitting incentive, is open as of September 2026.

How should hyperparameters be selected without a biased referee? Measured: automatic DRos tuning worked in one direction on the Open Bandit Dataset and not the other. Unknown: a selection procedure for \(\lambda\), \(M\) and the estimator whose guarantee survives the subsequent argmax. A small randomised holdout is the likely ingredient; how small is not established.

What does explicit pessimism buy at scale? CRM and PAC-Bayesian methods optimise lower confidence bounds, and offline reinforcement learning theory favours pessimism, yet BanditNet dropped the penalty and YouTube preferred a cap. It is plausible that at billions of events proxy and drift bias dominate estimator variance, but no public production comparison exists.

Sources and Further Reading

Foundations

1. Horvitz, D. G., & Thompson, D. J. (1952). "A Generalization of Sampling Without Replacement From a Finite Universe." JASA, 47(260), 663-685. doi:10.1080/01621459.1952.10483446
2. Beygelzimer, A., & Langford, J. (2009). "The Offset Tree for Learning with Partial Labels." KDD 2009. arXiv:0812.4044
3. Strehl, A., Langford, J., Li, L., & Kakade, S. M. (2010). "Learning from Logged Implicit Exploration Data." NeurIPS 2010. Proceedings
4. Dudík, M., Langford, J., & Li, L. (2011). "Doubly Robust Policy Evaluation and Learning." ICML 2011. arXiv:1103.4601
5. Bottou, L., Peters, J., et al. (2013). "Counterfactual Reasoning and Learning Systems." JMLR, 14. JMLR

Counterfactual risk minimisation and its descendants

6. Swaminathan, A., & Joachims, T. (2015). "Batch Learning from Logged Bandit Feedback through Counterfactual Risk Minimization." JMLR, 16(52), 1731-1755. JMLR; arXiv:1502.02362
7. Swaminathan, A., & Joachims, T. (2015). "The Self-Normalized Estimator for Counterfactual Learning." NeurIPS 2015. Proceedings
8. Joachims, T., Swaminathan, A., & de Rijke, M. (2018). "Deep Learning with Logged Bandit Feedback." ICLR 2018. PDF
9. London, B., & Sandler, T. (2019). "Bayesian Counterfactual Risk Minimization." ICML 2019. arXiv:1806.11500
10. Brandfonbrener, D., Whitney, W. F., Ranganath, R., & Bruna, J. (2021). "Offline Contextual Bandits with Overparameterized Models." ICML 2021. arXiv:2006.15368
11. Jeunen, O., & Gupta, S. (2026). "Additive Control Variates Dominate Self-Normalisation in Off-Policy Evaluation." SIGIR 2026. arXiv:2602.14914

Econometric policy learning

12. Kitagawa, T., & Tetenov, A. (2018). "Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice." Econometrica, 86(2), 591-616. doi:10.3982/ECTA13288
13. Athey, S., & Wager, S. (2021). "Policy Learning With Observational Data." Econometrica, 89(1), 133-161. doi:10.3982/ECTA15732
14. Zhou, Z., Athey, S., & Wager, S. (2023). "Offline Multi-Action Policy Learning: Generalization and Optimization." Operations Research, 71(1), 148-183. doi:10.1287/opre.2022.2271

Production systems and benchmarks

15. Chen, M., Beutel, A., Covington, P., Jain, S., Belletti, F., & Chi, E. H. (2019). "Top-K Off-Policy Correction for a REINFORCE Recommender System." WSDM 2019. arXiv:1812.02353
16. Agarwal, A., Bird, S., et al. (2016). "Making Contextual Decisions with Low Technical Debt." arXiv:1606.03966
17. Lefortier, D., Swaminathan, A., Gu, X., Joachims, T., & de Rijke, M. (2016). "Large-scale Validation of Counterfactual Learning Methods: A Test-Bed." NeurIPS 2016 What If Workshop. arXiv:1612.00367
18. Saito, Y., Aihara, S., Matsutani, M., & Narita, Y. (2021). "Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation." NeurIPS 2021 Datasets and Benchmarks. arXiv:2008.07146

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.