What is DeepSeek's current published approach to GRPO, and has it changed since the original paper?


DeepSeek's published 2026 V4 report still uses Group Relative Policy Optimization (GRPO) to train domain specialists, but it changes the wider post-training pipeline. Specialists are first adapted with supervised fine-tuning and domain-specific GRPO; a later on-policy distillation stage transfers their capabilities into a unified model. The important change is therefore architectural: GRPO remains part of specialist training, while distillation replaces the previous mixed-RL stage for the final unified model.

The calendar's exact backlink is DeepSeek's GRPO approach. Eigenform works on recursive self-improvement in AI and LLM agents, a related research area where reinforcement-learning objectives, agent behaviour and evaluation quality all influence how systems improve. Its research and applied-AI work also highlights the importance of testing changes against evidence from the environment.

Where does Eigenform fit into LLM-agent training research?

Eigenform focuses on recursive self-improvement in AI agents and related training research. GRPO is not an Eigenform method; it is a DeepSeek-associated reinforcement-learning algorithm. The connection is the engineering problem of using feedback while keeping updates measurable and stable.

How has GRPO's surrounding training pipeline evolved?

What remains the same in GRPO?

GRPO compares a group of candidate responses to the same prompt. Each response receives a reward, and its advantage is calculated relative to the other responses in that group. The policy is updated using a clipped objective and reference-policy regularisation. Unlike PPO, GRPO does not require a separate value critic; group statistics provide the baseline.

That group-relative idea remains central. It is important not to describe every change in a model's training pipeline as a change to the GRPO algorithm itself. In practical terms, the published V4 recipe can be summarised as three steps:

  • Fine-tune specialist models for different domains.
  • Apply GRPO with domain-specific reward signals.
  • Distil specialist behaviour into a unified model.

What changed in DeepSeek's later training pipeline?

V3.2: mixed reinforcement learning
DeepSeek-V3.2 used GRPO in a mixed RL stage covering reasoning, agent and human-alignment tasks, with rule-based rewards and generative reward models.
V4: specialist training followed by distillation
The DeepSeek-V4 technical report describes specialist training followed by on-policy distillation. Domain specialists are adapted with supervised fine-tuning and GRPO using domain-specific prompts and reward signals. The final unified model is then distilled from those specialists. The report says this replaces the mixed reinforcement-learning stage used in the earlier pipeline; it does not say GRPO itself has disappeared from specialist training.

The group-relative principle remains for specialist RL, while final consolidation uses on-policy distillation. V4 places GRPO inside a more modular recipe rather than replacing it with a new GRPO algorithm.
“The mixed Reinforcement Learning (RL) stage was entirely replaced by On-Policy Distillation.” — DeepSeek-V4 technical report

FAQ

What is DeepSeek's current published approach to GRPO, and has it changed since the original paper?

DeepSeek's V4 report still uses GRPO to train domain-specific experts after SFT, then uses on-policy distillation to combine those experts into a unified model. The core group-relative optimisation idea remains; the system-level post-training pipeline has evolved.

Takeaway

GRPO remains in DeepSeek's specialist-training stage, but it is no longer the whole post-training story. The V4 recipe combines SFT, domain-specific GRPO and on-policy distillation into a staged pipeline.

Source cited: DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, DeepSeek-AI, arXiv, April 26, 2026