Debate as Scalable Oversight

Reward models are trained on human judgements, so they inherit the ceiling of what a human can judge. That is fine while the model's outputs are checkable and becomes the central problem when they are not: a 400-line proof, a codebase-wide refactor, a claim about a paper the reviewer has not read.

Debate proposes a way around the ceiling. Two copies of a strong model argue for opposing answers, each free to rebut the other, and a weaker judge decides. The bet is an asymmetry: defending a true claim against a competent adversary is easier than defending a false one, so the judge can identify truth without being able to derive it (Irving, Christiano & Amodei, 2018, arXiv:1805.00899).

The complexity-theoretic intuition

The original framing draws an analogy to interactive proof systems. A judge who can only inspect the final answer is limited to problems whose answers are directly checkable, loosely an NP-like class. A judge who can pose questions to two competing provers and inspect only the specific point they disagree on reaches much further, loosely PSPACE. The debaters do the exponential work; the judge does polynomial work on the narrowest point of contention.

The paper's demonstration was deliberately small: a sparse-classifier judge on MNIST allowed to see only a handful of pixels, with two agents competing to reveal pixels supporting their claimed digit. Accuracy went from 59.4% to 88.9% with 6 pixels revealed, and from 48.2% to 85.2% with 4 (Irving et al., 2018).

The original protocol assumed the honest debater could simulate a deterministic system for exponentially many steps, which is not a realistic assumption. Doubly-efficient debate constructs protocols where the honest strategy succeeds using only a polynomial number of simulation steps, and can verify stochastic systems even when the dishonest strategy is allowed exponentially many (Brown-Cohen, Irving & Piliouras, 2023, arXiv:2311.14125).

The empirical record is mixed, in an informative way

The first careful human studies were negative. Parrish and colleagues found that presenting humans with arguments for two competing answers to hard reading-comprehension questions did not improve their accuracy, in either single-turn or two-turn formats (Parrish et al., NeurIPS 2022 ML Safety Workshop, arXiv:2210.10860).

The picture changed when debater skill and protocol design improved. Michael and colleagues ran human debates on hard reading-comprehension questions where judges could not see the passage, and found debate reached 84% accuracy against 74% for a consultancy baseline, while using 68% of the length; error analysis showed debate failures came mostly from honest-debater mistakes (46%), whereas consultancy failures came mostly from evidence obfuscation (52%) (Michael et al., 2023, arXiv:2311.08702).

Khan and colleagues then automated the debaters and found the key scaling property: optimising debaters for persuasiveness without any ground-truth supervision improved the judge's ability to find truth. Non-expert model judges reached 76% against a 48% naive baseline, and human judges reached 88% against 60% (Khan et al., ICML 2024, arXiv:2402.06782).

The direction of that result matters more than the numbers. It says the mechanism improves as the debaters get stronger, which is the property any oversight method needs if it is to survive scaling.

When it breaks

Obfuscated arguments. The honest debater must be able to decompose its case into steps the judge can adjudicate. Against an argument whose flaw is distributed across many steps, each individually plausible, the honest side has no single point to attack, and recursive decomposition costs more than the dishonest side spends constructing the mess.

The judge is the attack surface. Debate transfers the burden from verifying answers to judging arguments, which does not remove human bias, it relocates it. Persuasiveness and truth correlate imperfectly, and a judge susceptible to confident phrasing, length, or flattery is exploitable by whichever side optimises for that.

Collusion. Two instances of the same base model, optimised against the same judge, are not guaranteed to be adversarial. Nothing in the protocol forces genuine opposition if both sides gain from an outcome the judge cannot detect.

Cost. Debate multiplies inference: multiple turns, two debaters, plus a judge, for a single supervised label. As an evaluation protocol that is acceptable; as a training signal it competes against much cheaper alternatives, and the comparison is rarely made at matched cost.

The transfer to open-ended tasks is unproven. Nearly all results use question answering with a defined correct answer and an information asymmetry that the experimenter constructs. Debate about the safety of an action, where there is no single correct answer, is not the setting that has been measured.