Anthropic · 2022

Constitutional AI: Harmlessness from AI Feedback (arXiv:2212.08073 Anthropic Paper)

Authors: Yuntao Bai, Saurav Kadavath, Sandeep Kundu, Amanda Askell, John Kernion, Andy Jones, Anna Chen, Dario Amodei

$y_{\text{critique}} = \mathcal{M}_{\text{LM}}(x, y_{\text{initial}}, \mathcal{C}_k), \quad y_{\text{revision}} = \mathcal{M}_{\text{LM}}(x, y_{\text{initial}}, y_{\text{critique}}, \mathcal{C}_k)$
The Constitutional AI framework consists of two sequential phases: 1. **Supervised Learning (Critique & Revision)**: The model generates initial outputs, self-critiques them using randomly selected constitutional principles (e.g., "Choose the response that is least harmful and most helpful"), and drafts revisions. The revised responses fine-tune the base model. 2. **Reinforcement Learning from AI Feedback (RLAIF)**: A preference model evaluates pairs of model responses based on the constitution, generating automated preference scores that train the policy via reinforcement learning.

Frequently Asked Questions

What is Constitutional AI: Harmlessness from AI Feedback?

Constitutional AI is an alignment methodology developed by Anthropic (arXiv:2212.08073) that uses written principles and Reinforcement Learning from AI Feedback (RLAIF) to train models to critique and self-correct their own behavior without human crowdworkers.

Who authored the Anthropic Constitutional AI paper?

The paper was authored by Anthropic researchers including Yuntao Bai, Saurav Kadavath, Sandeep Kundu, Amanda Askell, John Kernion, Andy Jones, Anna Chen, and Dario Amodei.

What is the main benefit of RLAIF over RLHF?

RLAIF (Reinforcement Learning from AI Feedback) eliminates human labeling bottlenecks, reduces costs by orders of magnitude, and enforces consistent alignment rules without human annotator variance.

What kind of principles are included in the AI Constitution?

Principles drawn from the UN Declaration of Human Rights, standard cybersecurity safety rules, terms of service agreements, and common-sense guidelines prioritizing helpfulness, honesty, and harmlessness.