Multi-Agent RL
Self-play, equilibria, credit assignment across agents, emergent coordination and non-stationarity.
11concepts
99flashcards
80minutes of reading
- 01 Counterfactual Regret Minimisation How CFR splits the regret of a whole imperfect-information game into independent local regrets at each information set, why the average strategy converges to a Nash equilibrium only in two-player zero-sum games, and how CFR+, Linear CFR, Libratus and Pluribus turned the theory into superhuman poker.
- 02 Credit Assignment Across Agents Why a shared team reward gives every agent the same noisy signal, how counterfactual baselines and value factorisation isolate individual contributions, and the structural assumption each makes.
- 03 Dec-POMDPs and the Complexity of Decentralised Control The formal model behind cooperative multi-agent RL, why finite-horizon planning in it is NEXP-complete rather than merely intractable, and what that complexity result forces every practical algorithm to assume.
- 04 Emergent Coordination and Communication How agents develop coordinated behaviour without being told to, why learned communication protocols are hard to train and harder to interpret, and what counts as evidence of genuine communication.
- 05 Equilibria as Learning Targets Why Nash equilibrium is the wrong objective for most multi-agent learning, what the alternatives assume, and how the game's structure determines whether convergence is even possible.
- 06 Mean-Field Approximation for Many-Agent Learning How replacing every other agent with a population average turns an exponentially large joint action space into a two-body problem, and the exchangeability assumption that makes the trick either exact or badly wrong.
- 07 Non-Stationarity in Multi-Agent Learning Why every guarantee from single-agent RL dissolves when other learners are present, what breaks in the Markov assumption, and the two architectural responses.
- 08 Opponent Shaping and Learning-Aware Agents Why differentiating through another agent's learning step turns mutual defection into reciprocal cooperation, what access that requires, and how shaping becomes exploitation once both sides do it.
- 09 Self-Play and Population-Based Training Why playing against yourself generates an automatic curriculum, the cycling and forgetting failures that follow, and how a league of opponents fixes both.
- 10 Sequential Social Dilemmas and Mixed-Motive Learning Why embedding a cooperation dilemma in time and space changes what cooperation even is, how learned behaviour shifts with resource abundance, and what makes evidence about cooperation so easy to overstate.
- 11 Zero-Shot Coordination and Cross-Play Why agents trained by self-play reach near-perfect scores with their own copies and collapse with independently trained partners, what cross-play measures, and how symmetry-breaking explains the gap.