Scalable Oversight and Weak-to-Strong Generalisation

RLHF works because a human can look at two summaries and say which is better. Now replace the summaries with two 4,000-line pull requests, or two proofs, or two clinical protocols. The labeller is still human and the label is now noise. Every alignment technique that depends on human judgement has a ceiling set by human ability to judge, and models are crossing that ceiling task by task.

Scalable oversight is the research programme aimed at that ceiling: methods for supervising a system on tasks where the supervisor cannot directly verify the answer.

The weak-to-strong analogy

OpenAI's superalignment team proposed an experimental proxy that made the problem tractable today. Instead of a human supervising a superhuman model, fine-tune a strong model (say GPT-4) on labels produced by a much weaker one (GPT-2), and ask how much of the strong model's own capability survives (Burns et al., 2023, arXiv:2312.09390).

The headline result is that the strong student consistently outperforms its weak supervisor. It does not merely imitate the weak labels including their errors; pretraining has already given it the concept, and weak supervision partly elicits it. The metric they introduced is the performance gap recovered: the fraction of the distance between weak-supervisor performance and strong-ceiling performance that the student actually closes. Naive fine-tuning recovers a meaningful fraction on NLP tasks and much less on reward modelling. An auxiliary confidence loss, which encourages the student to make confident predictions even where they disagree with the weak label, recovered a substantially larger fraction on NLP tasks.

Eliciting is the right verb. The knowledge is in the weights; supervision is a pointer, not a teacher.

The other families

Debate. Two copies of a model argue for opposing answers and a weaker judge decides (Irving et al., 2018, arXiv:1805.00899). The bet is that finding a flaw in an argument is easier than constructing the argument, so a judge who cannot solve the problem can still adjudicate it. It works when the argument decomposes into checkable steps and degrades when the most persuasive argument is not the true one.

Recursive reward modelling and task decomposition. Break the unevaluable task into subtasks that are evaluable, and use model assistance at each level. The risk is that errors accumulate down the recursion faster than the decomposition helps.

Constitutional AI. Replace the human harmlessness label with a written principle plus a model critique, which is scalable oversight applied to one specific axis (Bai et al., 2022, arXiv:2212.08073). Covered separately in Constitutional AI and RLAIF.

Where the analogy leaks

The weak-to-strong setup is an analogy, and its authors say so. Three disanalogies matter.

The weak supervisor's errors are the wrong shape. GPT-2's mistakes are incoherent. A human supervisor's mistakes are systematic and correlated with human biases, which is exactly the kind of error a model can learn to exploit rather than correct.

The strong model already knows the task. Pretraining saw human-written text about these tasks. A genuinely superhuman capability, one with no representation in the pretraining corpus, has nothing to elicit.

Salience is not the same as capability. The student may be surfacing a representation that was already easy to reach. That says little about whether the same trick works when the correct behaviour is one the model has an incentive to hide.

When it breaks

Elicitation and alignment are different targets. Recovering capability from weak labels tells you the model can do the task. It does not tell you it will do the task honestly when the supervisor cannot check.

Judges get persuaded. Debate protocols assume the judge tracks argument quality. Empirically, LLM judges track fluency, length, and confidence, which is precisely what a deceptive debater optimises.

The oversight gap is measurable and widening. Scaling-laws work on oversight suggests the advantage of the overseer over the overseen shrinks as both scale, which is an argument for AI control as a complement rather than a competitor (Engels et al., 2025, Scaling Laws for Scalable Oversight, arXiv:2504.18530).