Off-Policy Evaluation for Rankers and LLM Systems

The theory of off-policy evaluation is clean and the systems it is applied to violate almost all of its preconditions. A ranker's action is an ordered list; an LLM's action is a token sequence. Both spaces are effectively unbounded, both logging policies are usually deterministic or near-deterministic, and both reward signals are partial. Understanding which assumption is broken decides which of the available techniques is worth trying.

Ranking: the slate problem

For a ten-item slate drawn from a large catalogue, the number of distinct actions exceeds anything the logs can cover, so the importance ratio \(\pi(a\mid x)/\pi_0(a \mid x)\) is almost always zero or astronomically large. Plain IPS is useless.

Every practical estimator buys tractability with a structural assumption about how the reward decomposes.

Position-based propensity. Assume the probability a user examines position \(k\) depends only on \(k\), so clicks factorise into examination times relevance. The examination curve can then be estimated, often by randomised result-swap interventions, and used as the propensity. This is the standard foundation of unbiased learning to rank.

Pseudoinverse and slate estimators. Assume the slate reward is a sum of per-position, per-item contributions. This makes the effective action space linear in slate length times catalogue size rather than exponential, at the cost of ruling out interaction between items in the list.

Both assumptions are about user behaviour, not about the algorithm, and they are wrong in specific known ways: users compare items within a list, and examination depends on the content above a position and not only on its index. Their value is that they are wrong in a bounded, arguable way rather than an unbounded one.

LLM systems: the propensities were never logged

For a generative system the situation is different in kind. Token-level probabilities exist, so a sequence propensity is in principle computable, but the product over hundreds of tokens has variance that makes it worthless, and in practice logs from a deployed assistant rarely retain them. Temperature-zero decoding makes the logging policy deterministic outright.

What is used instead is not off-policy evaluation in the importance-weighting sense. It is a different set of tools with different guarantees.

Replay against a fixed eval set with an automated judge estimates the new system's quality directly rather than reweighting old outcomes. It sidesteps the propensity problem entirely and inherits the judge's biases and the eval set's coverage instead.

Paired comparison on logged inputs runs both systems on the same historical prompts and compares outputs, which controls for input distribution but cannot observe how a user would have reacted to the new output. Anything downstream of the user's response, follow-up rate, task completion, retention, is unavailable.

Interleaving, borrowed from search, mixes both systems' outputs within a single session and attributes the user's engagement. It is a genuine online method with strong sensitivity, and it applies only where outputs can be meaningfully mixed.

The honest summary is that for generative systems the counterfactual question is usually answered by an online experiment, and offline methods serve to decide which candidates are worth the traffic.

When it breaks

Deterministic logging leaves no counterfactual. A production system serving argmax has no exploration, so no reweighting is possible. The structural fix is deliberate randomisation in production: epsilon-greedy exploration, Thompson sampling, or a small randomised traffic slice with propensities logged. This has to be built in before the evaluation question is asked, and it is a product cost, not just an engineering one.

Reward is partial and delayed. A click is observed, an absent click is ambiguous between "not relevant" and "not examined", and satisfaction arrives later if at all. Every estimator here assumes the logged reward is the reward.

Confidence intervals from importance weighting are usually understated. With effective sample size orders of magnitude below the raw log count, standard errors computed on the raw count are meaningless. Reporting ESS alongside every off-policy estimate is the minimum discipline.

Offline agreement does not imply online agreement. A candidate that wins offline and loses online is common enough to be the expected outcome rather than a surprise, because the offline estimate holds the user's behaviour fixed and deployment changes it. Offline evaluation is a filter, not a decision.