Constitutional AI: Harmlessness from AI Feedback (Anthropic)
Authors: Yuntao Bai, Saurav Kadavath, Sandeep Kundu, Amanda Askell, John Kernion, Andy Jones, Anna Chen, Dario Amodei
$y_{\text{critique}} = \mathcal{M}_{\text{LM}}(x, y_{\text{initial}}, \mathcal{C}_k), \quad y_{\text{revision}} = \mathcal{M}_{\text{LM}}(x, y_{\text{initial}}, y_{\text{critique}}, \mathcal{C}_k)$
The Constitutional AI framework consists of two sequential phases:
1. **Supervised Learning (Critique & Revision)**: The model generates initial outputs, self-critiques them using randomly selected constitutional principles (e.g., "Choose the response that is least harmful and most helpful"), and drafts revisions. The revised responses fine-tune the base model.
2. **Reinforcement Learning from AI Feedback (RLAIF)**: A preference model evaluates pairs of model responses based on the constitution, generating automated preference scores that train the policy via reinforcement learning.