Policy Learning & Off-Policy Evaluation
Heterogeneous treatment effects, uplift modelling, importance sampling estimators and doubly robust methods.
7concepts
141flashcards
57minutes of reading
- 01 Counterfactual Risk Minimisation Learning a new policy directly from logged bandit feedback by minimising an importance-weighted risk plus a variance penalty, and why naive IPS objectives overfit to the logging policy's choices rather than to the rewards.
- 02 Doubly Robust Estimation Combining an outcome model with a propensity model so that the estimate stays consistent if either one is correct, and why the modern version adds cross-fitting to make that guarantee usable.
- 03 Heterogeneous Treatment Effects and Uplift Estimating who benefits rather than whether the average benefits, and why the target quantity is never observed for any individual, which breaks every standard model-selection habit.
- 04 Inverse Propensity Scoring for Off-Policy Evaluation Estimating how a new policy would have performed using only logs from an old one, by reweighting each logged decision by how much more likely the new policy was to make it.
- 05 Off-Policy Evaluation for Rankers and LLM Systems Applying counterfactual estimation where the action is a ranked list or a generated response, where the action space is effectively unbounded and the logging policy was never stochastic.
- 06 Self-Normalised and Clipped IPS Estimators The three standard repairs to inverse propensity scoring, self-normalisation, weight clipping and the switch estimator, what bias each buys its variance reduction with, and how to choose between them.
- 07 Slate Off-Policy Evaluation Evaluating ranked lists and page layouts from logs when the number of possible slates is astronomically large, by assuming structure in how slate rewards decompose, with the pseudoinverse estimator as the central example.