Constitutional AI and RLAIF

RLHF works but it has a labour problem. Harmlessness preferences require humans to read disturbing prompts and rank disturbing outputs. The labellers burn out, the labels are inconsistent across demographics, and you cannot afford to relabel every time you tweak the policy. Constitutional AI (Bai et al, Anthropic 2022) replaces the harmlessness half of RLHF with a written set of principles and a model that critiques itself against them.

The two-phase training loop

Phase 1: Supervised - critique and revise.

Starting from an SFT-helpful model:

1. Sample a harmful prompt.
2. Generate an initial (potentially harmful) response.
3. Prompt the same model: "Identify ways in which the response is harmful, unethical, racist, etc. according to [principle from constitution]."
4. Prompt again: "Rewrite the response to remove the harmful content."
5. Fine-tune on (prompt, revised response) pairs.

You iterate this for several principles and several rounds. The principles are explicit text, e.g. "Please choose the response that is the most helpful, honest, and harmless" or "Choose the response that a wise, ethical, polite and friendly person would more likely say".

Phase 2: Reinforcement learning from AI feedback (RLAIF).

1. For each prompt, sample two responses from the SL-trained model.
2. Prompt a separate AI model with a constitutional principle and ask it to pick the better response.
3. Train a preference model on these AI-generated comparisons.
4. Run RL (PPO) against the preference model, with KL penalty against the SL model.

You have replaced the human pairwise comparisons in standard RLHF with model-generated ones. The reward model is now trained on AI labels, not human labels - hence RLAIF.

Why a written constitution matters

Standard RLHF encodes harmlessness implicitly in the weights of the reward model. If you ask "why did the model refuse that?" the answer is "because the reward model gave the refusal a higher score." That is not auditable.

CAI encodes harmlessness as text. The principles can be:

• Inspected (publish the constitution, debate the wording).
• Diffed across releases (this principle was added in v2 after observing X).
• Edited surgically (loosen this principle, tighten that one, rerun the loop).
• Stress-tested (does the model actually behave consistently with principle 7?).

Anthropic publish their working constitution. The principles include excerpts from the UN Declaration of Human Rights, Apple's terms of service, and DeepMind's Sparrow rules. The mixing is deliberate - it teaches the preference model what kind of cross-cultural, multi-perspective trade-off to make.

How it differs from and complements RLHF

In practice modern training pipelines mix both. Anthropic, OpenAI, Google, and Meta all use AI feedback for at least part of preference labelling now (sometimes called "RLAIF" generically, sometimes "synthetic preference data"). The pure-CAI recipe is less interesting than the separation of helpful labels from harmless labels and the use of written principles to anchor the critic.

What works, what doesn't

• Works. Drastically reducing the human-label burden for harmlessness. Making the policy auditable and editable. Letting safety teams iterate faster than human labelling allows.
• Partially works. The critic is the same model class as the policy - shared blind spots stay blind. Constitutional principles do not magically generalise; you still need red-teaming to find where they fail.
• Does not work. Treating the constitution as a substitute for capability evaluations. A model that scores well on RLAIF can still be jailbroken, can still exfiltrate data, can still hallucinate. CAI shapes refusal style; it does not bound underlying capability.

The unresolved question is recursive: if you use AI feedback to train models that will themselves provide AI feedback for stronger models, you risk amplifying the critic's blind spots across generations. Anthropic's later work on debate, scalable oversight, and the Alignment Faking finding all sit in this thread.

Further reading

• Bai et al, Constitutional AI - Harmlessness from AI Feedback (2022)
• Anthropic Responsible Scaling Policy