advanced practice · coding problem
GRPO group-relative advantage
Four completions with rewards [1.0, 0.5, 0.8, 0.3]. Mean μ=0.65, population std σ≈0.269. What is the normalized advantage for the top completion (1.0)?
The mechanism
Post-training RL treats the LM as a policy: sample completions, score them with a reward model, then nudge token probabilities toward high-reward outputs. GRPO compares completions within a group (relative advantages) and reuses rollouts with importance sampling + clipping so training stays stable off-policy.
Expected evidence
(1.0 − 0.65) / 0.269 ≈ 1.30 — GRPO upweights this rollout relative to the group.
Try this drill in the practice workspace →
Hints when you need them
- Advantage = (reward − mean) / std across the group.
- Same prompt, multiple sampled completions — compare relatively, not absolutely.