Self-Play and Population-Based Training

Self-play is the most striking idea in competitive multi-agent learning: an agent trained against copies of itself faces an opponent exactly at its own level, which improves as it improves. The curriculum is automatic and free, it requires no human data, and it produced superhuman play in Go, chess and shogi from random initialisation. It also has two failure modes that are easy to reproduce and that determine most of the engineering.

The two failures

Strategic cycling. In games without a dominant strategy, the best response to strategy A is B, to B is C, and to C is A. An agent training against its current self follows that cycle indefinitely, appearing to improve at every step while going nowhere. Win rate against the immediate previous version is high throughout, which makes the failure invisible to the most natural progress metric.

Catastrophic forgetting. An agent that has learned to counter a strategy stops encountering it once the opponent, which is itself, abandons it. The ability decays, the strategy becomes viable again, and the agent may relearn and re-forget it repeatedly.

Both have the same root: the opponent distribution is a single point that moves, so the agent optimises against a target rather than against a distribution.

Population-based training

The fix is to make the opponent distribution broad and to include history.

Play against past versions. Sampling opponents from a pool of earlier checkpoints prevents forgetting, because old strategies keep appearing. Fictitious self-play, playing the average of all past policies, has convergence properties for certain game classes that pure self-play lacks.

Maintain a diverse population. Several agents trained with different objectives, hyperparameters or reward shaping produce genuinely different strategies rather than variations of one.

Add exploiters. AlphaStar's league included agents whose objective was specifically to defeat the current main agent, which finds exploitable weaknesses deliberately rather than waiting for them to appear. This is the mechanism that most directly converts a brittle strategy into a robust one.

Prioritise opponent sampling by difficulty or by informativeness, so training time is spent on matchups that produce learning rather than on ones already won.

When it breaks

Compute cost multiplies. A league is many agents trained simultaneously plus the matchmaking between them, so the budget is a multiple of single-agent self-play. This is the main reason league approaches appear in well-resourced projects and rarely elsewhere.

Evaluation needs a fixed reference. Win rate against the current opponent measures relative performance in a moving population and can be flat while absolute skill rises, or high while it stalls. A frozen benchmark opponent, or Elo computed over a fixed pool, is required to measure progress at all.

Self-play requires a symmetric competitive structure. It does not apply to cooperative tasks, to asymmetric games without careful role handling, or to any setting where the environment is not itself the opponent. Applying it outside that structure produces training that does not correspond to the deployment condition.

Superhuman in self-play is not superhuman generally. An agent optimised against a population of its own descendants is strong against those strategies specifically, and exploitable strategies outside that distribution have repeatedly been found by humans against systems that dominate their own leagues.