DeepSeek-R1: Emergence of Reasoning via Pure Reinforcement Learning
Authors: DeepSeek-AI Team, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu
$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(O|q)} \left[ \frac{1}{G} \sum_{i=1}^G \min\left( \frac{\pi_\theta(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)} \tilde{A}_i, \text{clip}\left(\frac{\pi_\theta(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)}, 1-\epsilon, 1+\epsilon\right) \tilde{A}_i \right) \right]$
Key technical innovations in DeepSeek-R1:
1. **DeepSeek-R1-Zero**: Trained directly from base model weights without human demonstrations. Discovered an emergent "aha moment" where the model re-read its own intermediate thoughts and corrected logical flaws.
2. **Elimination of the Critic Model**: Instead of maintaining a separate memory-heavy Value network (as in standard PPO), GRPO samples a group of candidate responses per question and normalizes rewards relative to the group mean and standard deviation.
3. **Multi-Stage Distillation**: Distilled the reasoning capabilities of the 671B model into smaller dense architectures (1.5B, 7B, 14B, 32B), allowing lightweight models to outperform older 70B models on math benchmarks.