Deploying Bandits in Production
The gap between the bandit formalism and a real system, the four assumptions that fail first, and what a deployment needs that the algorithm does not describe.
The bandit formalism assumes an action is taken, a reward arrives, and the next round begins. Production has none of that structure, and most of the work in a bandit deployment is reconciling the algorithm with the ways reality departs from its assumptions.
The four assumptions that fail
Immediate reward. Real outcomes arrive minutes to weeks later. A purchase attributed to a recommendation may occur the next day; a subscription renewal months later. The algorithm must act on incomplete information continuously, which favours methods that degrade gracefully under delay, and it must handle a reward arriving for a decision made under a model that has since changed.
Unambiguous attribution. A user sees ten recommendations and buys one item. Which decisions get credit is a modelling choice with no correct answer, and the choice determines what the bandit optimises. Getting attribution wrong is a more common cause of a bandit performing badly than any algorithmic issue.
Stationarity. Content ages, users change, seasons turn, and competitors act. An algorithm that has converged on a stationary assumption is confidently wrong after a shift, so discounting, sliding windows, or explicit change detection are required rather than optional.
One decision at a time. Systems make thousands of decisions per second, and feedback for none of them has returned. Batch and delayed-update variants are what handle this, and deterministic algorithms behave particularly badly here.
What a deployment needs
Propensity logging. The probability of each chosen action, recorded with the decision. This is what makes off-policy evaluation possible, so a candidate policy can be assessed on logged data before it is exposed to traffic. Without it, every change costs a live experiment.
A floor on exploration. Complete convergence means no data on unchosen actions, so the system cannot detect that one has improved. A small permanent exploration budget, often a fraction of a percent, keeps the log informative.
Guardrails and eligibility rules. The optimiser will exploit whatever it is rewarded for, so constraints on what may be shown, and to whom, belong outside the algorithm as hard filters rather than as reward penalties.
Cold-start handling. New arms have no data and, under most algorithms, high uncertainty and therefore high exploration priority, which can flood traffic to untested items. Explicit ramp limits on new arms are usually necessary.
When it breaks
Feedback loops narrow the data. The bandit's choices determine what is observed, so an arm rarely shown is rarely evaluated and stays rarely shown. Exploration is what breaks this, and it is the first thing cut when someone optimises short-term reward.
Reward hacking is available to a bandit too. Optimising click-through produces clickbait; optimising session length produces engagement patterns nobody wanted. The metric becomes the objective, and Goodhart's law applies with the speed of an online optimiser.
Interference between users breaks independence. In a marketplace, showing an item to one user affects its availability for others, so the arms are not independent and the regret guarantees do not hold. This is the same interference problem that complicates A/B testing, arriving in a setting with less statistical machinery to handle it.
Bandits are often the wrong tool. Where the decision is infrequent, the reward is very delayed, or a simple A/B test would answer the question, a bandit adds machinery and operational risk for little gain. They are worth it for high-volume, fast-feedback, many-option decisions, and are frequently applied well outside that.
14 flashcards for this concept
Click a card to reveal the answer.