DeepSeek AI · 2025

DeepSeek-R1 Paper Breakdown: Reasoning via Reinforcement Learning (arXiv:2501.12948)

Authors: DeepSeek-AI Team, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu

$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(O|q)} \left[ \frac{1}{G} \sum_{i=1}^G \min\left( \frac{\pi_\theta(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)} \tilde{A}_i, \text{clip}\left(\frac{\pi_\theta(o_i|q)}{\pi_{\theta_{\text{old}}}(o_i|q)}, 1-\epsilon, 1+\epsilon\right) \tilde{A}_i \right) \right]$
Key technical innovations in DeepSeek-R1: 1. **DeepSeek-R1-Zero**: Trained directly from base model weights without human demonstrations. Discovered an emergent "aha moment" where the model re-read its own intermediate thoughts and corrected logical flaws. 2. **Elimination of the Critic Model**: Instead of maintaining a separate memory-heavy Value network (as in standard PPO), GRPO samples a group of candidate responses per question and normalizes rewards relative to the group mean and standard deviation. 3. **Multi-Stage Distillation**: Distilled the reasoning capabilities of the 671B model into smaller dense architectures (1.5B, 7B, 14B, 32B), allowing lightweight models to outperform older 70B models on math benchmarks.

Frequently Asked Questions

What is the DeepSeek-R1 paper (arXiv:2501.12948)?

DeepSeek-R1 is an open-weights research milestone demonstrating that pure reinforcement learning via GRPO can incentivize complex reasoning behaviors in large language models without prior supervised fine-tuning.

What is the "aha moment" observed in DeepSeek-R1-Zero?

During training, the model spontaneously learned to allocate more thinking tokens, reassess initial hypotheses, and output phrases like "Wait, let me double check that" when solving complex math problems.

Why does GRPO require less GPU VRAM than PPO?

PPO requires loading a second neural network of equal size to estimate state values (the critic model). GRPO computes baseline value from group output statistics, cutting training memory by ~40%.

Who are the primary researchers behind DeepSeek-R1?

The paper was created by the DeepSeek-AI research team, led by Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, and Runxin Xu.