Post-Training RL & Verifiable Alignment
From RLHF to GRPO: Direct Preference Optimization, Group Relative Policy Optimization, and Self-Play.
Pillar Architectural Overview
### The New Frontier of Post-Training Optimization
Post-training has evolved from simple conversational alignment to mathematical policy optimization. Algorithms like Group Relative Policy Optimization (GRPO) eliminate the need for a separate critic model, computing baseline rewards directly from a group of model outputs.
$\nabla_{\theta} J(\theta) = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^G \frac{r_i - \bar{r}}{\sigma_r} \nabla_\theta \log \pi_\theta(y_i | x) \right]$
This formulation reduces training memory overhead by over 40% and directly incentivizes structural exploration.