Zero-Shot Coordination and Cross-Play

In two-player Hanabi, a state-of-the-art self-play agent scores 23.97 out of 25 when paired with a copy of itself. Pair it with an agent trained by the identical algorithm on a different random seed and the score falls to 2.52 (Hu et al., 2020, "Other-Play" for Zero-Shot Coordination, arXiv:2003.02979). Nothing about the task changed and nothing about either agent's skill changed. What changed is that the two agents had never agreed on what a hint means.

Self-play learns conventions, not skill

Many cooperative tasks have symmetries: relabellings of actions or observations that leave the payoff structure identical. In Hanabi, which colour a "hint colour" action refers to is arbitrary in the sense that any consistent assignment of meaning works equally well. Self-play optimises a single joint policy, so it is free to pick any one of these equivalent conventions, and gradient descent picks whichever the seed happened to favour.

The toy case in the Other-Play paper makes the mechanism exact. Two players independently pick one of ten levers; matching earns the lever's payoff. Nine levers pay 1.0 and one pays 0.9. Self-play converges on one of the nine identical high-payoff levers and scores 1.0 against itself. Against an unknown partner trained the same way, the expected return is about 0.11, barely better than the 0.1 from random choice, because there is no way to agree on which of the nine. The 0.9 lever is worse in self-play and strictly better in cross-play, because it is the only one with a unique label.

Cross-play is the metric

Self-play score evaluates a policy against a copy of itself. Cross-play evaluates agent \(i\) from one training run against agent \(j\) from an independent run of the same algorithm, averaged over pairings. The gap between them is the part of the self-play score that comes from private convention rather than competence, and it can be almost all of it.

Two methods close the gap in different places. Other-play maximises expected return when the partner's policy has been relabelled by a randomly drawn element of the known symmetry group, so a convention that depends on an arbitrary labelling stops paying. In Hanabi this lifted cross-play from 2.52 to 15.32, and to 22.07 when combined with an auxiliary task, with self-play essentially unchanged at about 24 (Hu et al., 2020). Off-belief learning goes further and removes conventions by construction: an agent is trained to act optimally under the belief that its partner played a fixed, convention-free baseline policy up to now, so no secret meaning can be attached to past actions. Iterated to level 4 it reaches 24.10 self-play and 23.76 cross-play with itself (Hu et al., 2021, Off-Belief Learning, ICML).

Three problems people conflate

Zero-shot coordination gives you no interaction with the partner before or during play beyond the game itself; the policy is fixed in advance and must work on first contact.

Ad hoc teamwork permits adaptation during play, and the question becomes how fast an agent can identify and accommodate a partner from observed behaviour (Stone et al., 2010, Ad Hoc Autonomous Agent Teams, AAAI).

Human-AI coordination is zero-shot coordination where the partner distribution is humans, which is not symmetric under relabelling, because human conventions are not arbitrary. An agent that is robust to arbitrary relabellings is not thereby good with people.

When it breaks

The symmetry group has to be known. Other-play needs the problem's symmetries specified by the designer. Where they are unknown, misstated, or only approximate, the method optimises for the wrong invariance.

Low cross-play is not always a convention problem. Re-examining Overcooked found that coordination failures there are driven largely by poor state coverage: partners drive the game into states the agent never saw in training, so the policy is simply out of distribution (Gessler et al., 2025, OvercookedV2, ICLR). Diagnosing a cross-play gap as a convention mismatch when it is a coverage gap sends you to the wrong fix.

Humans do not score it the way the benchmark does. In a single-blind study, people paired with an other-play Hanabi agent and with a hand-coded rule-based bot achieved no statistically different game score, yet preferred the rule-based partner on nearly every subjective measure including trust and interpretability (Siu et al., 2021, NeurIPS, arXiv:2107.07630).

Convention-free is not convention-competent. Human pairs exploit implicit meaning heavily. Measured as the gap between failure rates predicted from a hint's literal content and those observed, human-human play shows about +26.2 points of implicit communication while AI-AI play shows about -0.7 across roughly 101,000 replayed Hanabi actions (Fukushima et al., 2026, The Convention Gap, arXiv:2609.11489). Removing conventions solves cross-play by giving up the thing that makes human teams good.