Sequential Social Dilemmas and Mixed-Motive Learning

The prisoner's dilemma gives each player two buttons and one decision. Real cooperation problems give you a policy: whether to cooperate is not a choice but a temporally extended strategy that can be executed well or badly, abandoned halfway, or made conditional on what the other agent did ten steps ago. A sequential social dilemma (SSD) is the formalisation of that difference, and it changes the conclusions.

What the formalism adds

Leibo and colleagues define an SSD as a Markov game whose policies can be partitioned into cooperative and defecting classes such that the induced payoff matrix over those classes satisfies the inequalities of a matrix-game dilemma (Leibo et al., 2017, Multi-agent Reinforcement Learning in Sequential Social Dilemmas, AAMAS, arXiv:1702.03037). The dilemma structure is preserved at the level of strategies, and everything else is now a learning problem.

Their two environments are instructive in opposite directions. In Gathering, agents collect apples and can tag each other, temporarily removing a rival. Independent deep Q-learners play peacefully when apples are plentiful and turn aggressive as the resource becomes scarce: the same reward function, the same algorithm, different environmental abundance, opposite social behaviour. In Wolfpack, two agents hunt prey and are rewarded more for a joint capture, so the incentive runs the other way and cooperation is what has to be learned and sustained.

The lesson is that in an SSD, "cooperative" and "defecting" are not labels on actions. They are learned policies whose relative value depends on the environment's parameters, so the question "do agents cooperate?" has no answer without stating the abundance, the horizon, and the cost of defection.

The two mechanisms that produce cooperation

Changing the agents. Intrinsic motivation reshapes an individual's objective so that cooperation becomes locally rational. Inequity-averse agents, which penalise advantageous and disadvantageous payoff differences against their peers, sustain cooperation in intertemporal dilemmas where selfish learners do not, and the paper's analysis attributes the gain largely to better temporal credit assignment: a visible reward gap arrives sooner than the delayed collective payoff does (Hughes et al., 2018, NeurIPS, arXiv:1803.08884).

Changing the game. Adding enforcement, reputation, communication, or institutions alters the payoff structure until the dilemma is weaker or absent. This reliably produces cooperation, and it also means you are no longer studying the dilemma you started with. Any paper reporting cooperation should be read for which of the two it did.

Why evaluation is the hard part

Training performance says almost nothing here, because self-play in a mixed-motive game optimises against a co-player distribution of one's own copies. Melting Pot was built to address exactly this: substrates are paired with scenarios that insert held-out background populations, so a focal agent is scored against co-players it never trained with, across more than 80 test scenarios covering reciprocity, resource sharing, free-riding and trust (Leibo et al., 2021, Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot, ICML, arXiv:2107.06857). The suite's own finding is that it exposes weaknesses not visible from training performance.

When it breaks

Cooperation among identical copies is weak evidence. A population of clones trained together shares a policy, a reward shaping and a convention. It tells you little about behaviour against a population that does not, which is the case that matters for deployment and for the analogy to human institutions.

Reward shaping that buys cooperation changes the object of study. Inequity aversion, altruism terms and shaped social rewards all work by making the dilemma less dilemma-like. Reporting the resulting cooperation as a property of the learning algorithm overstates the result.

The cooperative/defecting partition is a modelling choice. The SSD definition requires a partition of policy space, and in rich environments there is no canonical one. Different partitions can turn the same Markov game into a dilemma or not.

Analogies to human societies outrun the evidence. These environments have small populations, no language, no enforcement institutions and reward functions chosen by a designer. They are good for studying learning dynamics under mixed motives and poor for conclusions about economics or policy.

Measures of cooperation are not comparable across papers. Collective return, equality, sustainability and peace are all reported as "cooperation", and an intervention can improve one while degrading another. Ask which scalar is being maximised before comparing two results.