Deliberative Alignment

Standard safety training gives the model examples of good and bad responses and lets it infer the rule. The model never sees the rule. It learns a decision boundary from labelled points, which means it generalises the way any classifier does: well near the training distribution, unpredictably outside it, and with no way to explain what it thinks the policy says.

The consequences are the familiar pair. Jailbreaks find inputs on the far side of a boundary that was never specified. Overrefusals happen when a superficially similar but benign request lands on the wrong side. Both are symptoms of the same gap: the model was taught a boundary rather than a rule.

Deliberative alignment closes it by writing the rule down and training the model to reason over it (Guan et al., 2025, Deliberative Alignment: Reasoning Enables Safer Language Models, arXiv:2412.16339).

The method

Two stages, and the second one is where the leverage is.

Stage one: supervised fine-tuning on specification-grounded reasoning. Generate chain-of-thought completions for prompts using a model given the relevant safety specification in context, then filter with a reward model that also has the specification. Strip the specification from the resulting training examples, keeping only prompt, chain of thought, and answer. The model is trained on (prompt → CoT → answer) with no policy text in the input, so it must internalise the specification's content into weights rather than reading it at inference.

Stage two: reinforcement learning with a specification-aware judge. A reward model that has access to the safety specification scores the outputs, sharpening the use of the reasoning. Crucially, no human-written chain of thought and no human-written response is used anywhere in the pipeline; the training data is model-generated and specification-filtered.

At inference the model recalls the relevant policy text inside its reasoning block, applies it to the specific request, and answers. This differs from Constitutional AI, which uses a written constitution to generate preference labels for training but does not teach the model to reason over the constitution at inference time (Bai et al., 2022, arXiv:2212.08073).

Why it improves both sides at once

The reported result that matters is a Pareto improvement: higher robustness to jailbreaks and lower overrefusal, on the o-series models it was used to align. Safety methods usually trade these against each other, since any monotone tightening of a decision boundary moves both in the same direction.

The mechanism that breaks the trade-off is that a rule is not a boundary. A model that can articulate "this policy prohibits synthesis instructions for controlled substances, and this request is a chemistry-history question" is doing case analysis rather than similarity matching. Case analysis generalises to inputs the boundary never covered, which is why the reported gains are largest on out-of-distribution safety evaluations.

The paper also reports improved generalisation to jailbreak styles not present in training, which is the property that most distinguishes this from adversarial-example-style safety training.

When it breaks

It depends on the specification being right. The method faithfully transmits whatever the policy document says, including its gaps, its ambiguities, and its errors. Specification quality becomes the dominant safety variable, which shifts the problem from an ML problem to a policy-drafting problem. That is arguably progress, since a written specification is auditable and a decision boundary is not.

Reasoning about the policy is not the same as being governed by it. The chain of thought is an intermediate the model produces, and there is no guarantee it is the actual cause of the answer. If the stated reasoning and the effective computation diverge, deliberative alignment produces a well-argued rationalisation of whatever the model was going to do. This is exactly the concern of chain-of-thought faithfulness, and it is unresolved.

It costs thinking tokens on every safety-relevant request, and the model must decide which requests are safety-relevant, which is itself a classification with the failure modes above.

Exposed reasoning is an attack surface. A model that recites its safety policy in a visible chain of thought is telling an attacker exactly what rules to route around. Hiding the reasoning mitigates the leak and removes the auditability that motivated the approach, which is a genuine and currently unresolved tension. See scalable oversight and weak-to-strong and injection-aware prompt design.