The Constitutional AI Data Loop

Anthropic's Claude models were trained on a feedback signal that was, in large part, written by Claude itself. Roughly 96 principles - the "constitution" - governed what the model was allowed to say, and the labelling work that would ordinarily require thousands of human annotators was delegated back to the model under instruction. That is the Constitutional AI (CAI) loop: a closed synthetic data pipeline where the policy model is simultaneously the student, the critic, and most of the grader.

The Two-Stage Architecture

CAI separates training into two sequential stages, each producing a distinct dataset.

Stage 1 - Supervised learning from revisions (SL-CAI). The initial model (call it M_0) is shown a harmful or borderline prompt. It generates a response, then is immediately prompted to critique that response against a randomly sampled constitutional principle - for example, "Identify specific ways in which the assistant's last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal." The model produces a critique, then a revised response. That (prompt, final-revision) pair becomes a supervised fine-tuning example. After accumulating enough such pairs, M_0 is fine-tuned to produce M_SL. The supervision signal is pure self-play; no human labeller sees the examples.

Stage 2 - Reinforcement learning from AI feedback (RL-CAI / RLAIF). M_SL generates pairs of responses to each prompt. A separate "feedback model" (in practice, a larger frozen LM) is prompted to choose which response better satisfies a constitutional principle framed as a multiple-choice question: "Which of these responses is less harmful?" Those AI preference labels train a preference model (PM). M_SL is then optimised against the PM reward via Proximal Policy Optimisation, yielding the final M_RL.

Prompt
  └─► M_0  ─── critique(principle) ──► revised response
                                              │
                       SFT on (prompt, revision) pairs
                                              │
                                          M_SL
                                              │
              M_SL generates response_A, response_B
                                              │
         Feedback LM labels: "A is better" / "B is better"
                                              │
                          Train preference model PM
                                              │
                        RL (PPO): optimise M_SL against PM
                                              │
                                          M_RL

Why the Constitution Matters

The word "constitutional" is precise: the principles are explicit, versioned, and auditable. This contrasts with RLHF, where human preferences are tacit and inconsistent across annotators. When a labeller chooses response A over B, you cannot inspect their reasoning; when a principle says "prefer the response that is least likely to contain false information," that criterion is visible in the training pipeline.

This has two practical consequences:

1. Scalable oversight. A single list of principles, applied by a capable LM, can label millions of preference pairs. Harrison Lee et al. (2024) showed that RLAIF achieves comparable win rates to RLHF on summarisation and dialogue tasks, and in some settings outperforms it.

2. Alignment transparency. Teams can diff the constitution between model versions the same way they diff code. Behavioural changes become partially traceable to principle changes.

The principles themselves are not arbitrary. In the Anthropic paper they drew on the UN Declaration of Human Rights, Apple's terms of service, and DeepMind's Sparrow rules - a deliberate eclecticism intended to avoid a single cultural or organisational bias dominating the signal.

How the Data Loop Compares to Other Synthetic Pipelines

Rejection sampling is the closest relative. Both methods use the model's own outputs as training data, but rejection sampling discards low-quality samples by a reward threshold rather than rewriting them. CAI rewrites: the critique-and-revise step produces a better response from the same prompt rather than simply filtering bad ones out.

The Critique-Revise Cycle in Detail

The SL-CAI revision chain is not a single step. Anthropic's paper ran up to four critique-revise iterations per prompt before treating the final response as the training example. Each cycle adds a marginal safety improvement but with diminishing returns; after roughly two or three iterations the response plateaus.

The principle is sampled randomly from the constitution at each step. This means the chain can look like:

Iteration 1: "Identify any racism or sexism in the response."
  → critique + revision_1

Iteration 2: "Does the response respect the user's autonomy?"
  → critique + revision_2

Iteration 3: "Would this response be considered harmful by a careful reader?"
  → critique + revision_3

Each revision sees the previous one as context, so the chain has memory. The randomness in principle selection prevents the policy from over-fitting to a narrow behavioural axis.

When It Falls Down

Reward over-optimisation. The preference model is itself an imperfect proxy for the true constitutional objectives. As PPO pushes the policy to maximise PM reward, it will eventually find responses that score well on the PM but violate the spirit of the principles. This is the classic Goodhart problem, and CAI does not escape it. Anthropic observed degradation in response quality at high KL-divergence from the supervised checkpoint.

Constitutional ambiguity. Principles can conflict. "Be helpful" and "refuse dangerous requests" pull in opposite directions on many prompts. The model's adjudication of that conflict during critique is not guaranteed to be consistent, and the inconsistency accumulates into the training data.

Feedback model bias. The feedback model used to label preference pairs inherits its own biases. If it prefers verbose responses, the PM will reward verbosity, and the policy will drift verbose. This is the same sycophancy risk present in any RLHF pipeline, now one step removed.

Self-consuming loop and diversity collapse. When the same model family is used for both policy and feedback, the training loop can reinforce the model's prior distribution rather than correct it. Alemohammad et al. (2023) demonstrated that iterative training on self-generated data ("autophagous loops") systematically degrades diversity over generations unless fresh human-authored data is injected at each round. A pure CAI pipeline with no external grounding is susceptible to exactly this failure mode.

Principle coverage gaps. The constitution is finite. Adversarial prompts designed to fall outside all stated principles will not be reliably handled. The constitution sets a ceiling on behavioural coverage equal to what its authors anticipated.

Further Reading

• Bai et al. (2022), "Constitutional AI: Harmlessness from AI Feedback" - the primary source: https://arxiv.org/abs/2212.08073
• Lee et al. (2024), "RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback" - empirical comparison of RLAIF and RLHF performance: https://arxiv.org/abs/2309.00267
• Wang et al. (2022), "Self-Instruct: Aligning Language Models with Self-Generated Instructions" - the broader instruction-synthesis context: https://arxiv.org/abs/2212.10560
• Alemohammad et al. (2023), "Self-Consuming Generative Models Go MAD" - formal analysis of diversity collapse in autophagous training loops: https://arxiv.org/abs/2307.01850