Self-Retrospection Distillation Lifts AI Agent Success From 0% to 60.6% When Every Rollout Fails

Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight

Authors: Haoxiang Zhang, Qinglin Chen, Hiroaki Hayashi, Zhuofeng Li, Siming Zhang, Jiaxin Zhang, Jixuan Chen, Fang Wu, Pan Lu, Silvio Savarese, Julian McAuley, Chien-Sheng Wu (Salesforce AI Research)

---

Abstract

Reinforcement learning with verifiable rewards (RLVR) turns agent experience into learning signals primarily through scalar outcome rewards after interaction. For group-relative objectives, however, this signal vanishes when all rollouts receive the same reward, even though their trajectories may reveal useful information about what the task requires and how the agent fails. We ask a complementary question: can hindsight teach an agent what it could have anticipated before acting? We introduce prospective learning, which uses post-hoc experience to supervise foresight predictions from the pre-interaction view, and instantiate it with Self-Retrospection Distillation (SRD). Intuitively, a completed trajectory reveals knowledge that would have been useful and pitfalls that should be avoided; SRD distills this privileged hindsight into trajectory-blind foresight of the same policy. Foresight serves only as a training target and need not be explicitly generated at inference time. Across 10 tool-integrated reasoning and long-horizon agentic tasks, SRD complements RLVR and self-distillation baselines with gains of up to 24.2 pp. Its advantage is especially pronounced when reward contrast is scarce: when 37–98% of rollout groups are reward-uniform across model scales, yet SRD can still exploit learning signal from sampled trajectories. In 2B setting, where 98% of groups are all-failure, the RLVR training ends up at 0.0% success, while adding SRD reaches 60.6% under the same rollout budget. Our results suggest that post-hoc agent experience is useful not only for evaluating or improving behavior, but also for shaping predictive representations before available interaction. Code

1 Introduction

Reinforcement learning with verifiable rewards (RLVR) has emerged as a scalable post-training paradigm for large language model (LLM) agents: by executing against automated verifiers and optimizing policies from outcome signals, it enables agents to develop complex, multi-step behavior without human-annotated traces (Shao et al., 2024; Guo et al., 2025; Yu et al., 2025). Applied to tool-integrated reasoning (Jin et al., 2025a; Xue et al., 2026; Qian et al., 2025), code execution (Le et al., 2022; Cao et al., 2026), computer use (Wang et al., 2025a), and multi-agent coordination (Li et al., 2026; Ke et al., 2026), RLVR has proven remarkably effective in settings where agents must act in an external environment and learn from feedback revealed only through interaction.

The supervision in RLVR, however, is inherently retrospective: outcome signals become available only after interaction and are then assigned back to the actions that produced them. Yet a broad line of work on predictive control, world models, and planning suggests that effective interaction also depends on what an agent can anticipate before acting, how its interaction may fail, and what the environment is likely to demand (Pathak et al., 2017; Hafner et al., 2020; Zhou et al., 2023). Such pre-interaction expectations shape subsequent exploration, but in RLVR they remain an implicit byproduct of optimizing behavior. More importantly, relying exclusively on post-hoc outcome contrast creates a fundamental blind spot: when all rollouts in a group receive the same reward, especially in long-horizon reasoning tasks where most attempts fail (Lightman et al., 2024), their group-relative advantages vanish (Schulman et al., 2017). The agent may have already revealed rich interactive information about the task and environment, yet learn no signal from that experience.

A recent line of on-policy self-distillation extracts a denser signal from this experience: a single model serves as both teacher and student, with the teacher conditioned on a privileged correct solution and the student trained to match its token-level distribution over the student’s own rollouts (Zhao et al., 2026; Hübotter et al., 2026). Follow-ups vary what the privileged context contains — runtime feedback, skill summaries, or self-revisions — and hybrid objectives fold this signal back into RLVR (Wang et al., 2026; Yang et al., 2026; He et al., 2026). Across these variants, however, the supervision target is unchanged: hindsight conditions the teacher to sharpen what the agent should do next. We ask instead what the agent could have known — whether the same completed experience can teach it to anticipate, before acting, the demands and failures that only interaction reveals.

We call this prospective learning: hindsight supervises the agent’s foresight. Given a task, the agent predicts from pre-interaction information alone the knowledge the task may require and the failures it may invite, and the completed trajectory then provides privileged supervision for those predictions.

Building on this principle, we propose Self-Retrospection Distillation (SRD). For each on-policy trajectory, SRD pairs a pre-interaction prediction with a hindsight prediction conditioned on the completed interaction, and aligns their token-level distributions on the student’s own prospective rollout. As a result, every trajectory can contribute prospective supervision, precisely where reward contrast leaves none: on a failed rollout, the self-teacher’s hindsight lets the agent distill the mistake it just made into an anticipation it can carry forward, and on a saturated all-correct group, the same pairing keeps extracting the knowledge that made those rollouts succeed. Reward thus decides which lesson a trajectory teaches rather than whether it teaches one; learning then continues at both ends of the difficulty range that group-relative methods discard. We make three contributions:

• A pre-interaction supervision target for agent experience. We formulate prospective learning (§3.1), a post-training target that distills structured information revealed after interaction into predictions constrained to the task and environment context available beforehand, complementary to the retrospective use of experience in RLVR and self-distillation.
• A composable self-distillation instantiation. Self-Retrospection Distillation (SRD, §3.2) adds a lightweight auxiliary objective to RLVR and self-distillation algorithms, and improves performance across diverse tool-integrated reasoning and long-horizon agentic settings (§4.2); our analysis characterizes the regimes in which this additional signal is most useful (§5).
• Learning beyond scalar reward variance. SRD provides dense supervision from all trajectories and remains informative for reward-uniform baselines where RL yields zero advantage (§5).

2 Preliminaries

We consider an agent policy πθ interacting with an environment \mathcal{E}. Given a task x\sim\mathcal{D}, the agent produces a trajectory τ=(a1,o1,…,aT,oT), where at∼πθ(⋅∣ht) is the action at step t, o_{t}\sim\mathcal{E}(\cdot\mid h_{t},a_{t}) the resulting observation, and ht the previous interaction history.

Reinforcement Learning with Verifiable Rewards (RLVR)

evaluates a completed trajectory with a verifier \mathcal{V} (e.g. a math checker, unit test, or task rubric), producing a scalar reward r(\tau)=\mathcal{V}(x,\tau) that is converted into an advantage estimate A⁡(τ) and assigned back to the actions that produced it:

\mathcal{L}_{\mathrm{RLVR}}(\theta)=-\mathbb{E}_{x,\,\tau\sim\pi_{\theta}}\Bigl[A(\tau)\sum_{t=1}^{T}\log\pi_{\theta}(a_{t}\mid h_{t})\Bigr]. 

For group-relative methods such as GRPO (Shao et al., 2024), given G rollouts {τi}i=1G for the same task with rewards {ri}i=1G, the advantage is A⁡(τi)=(ri−μr)/σr, where μr and σr are the group mean and standard deviation. Critically, when all rollouts receive identical rewards, A⁡(τi)=0 for every i; standard practice discards such uniform-reward groups entirely and resamples until a discriminative batch is obtained (Guo et al., 2025; Yu et al., 2025), prolonging the rollout and wasting trajectories and whatever information they contain.

Figure 1: Comparison of three paradigms for converting agent experience into training signals. Reinforcement Learning with Verifiable Rewards (RLVR) scalarizes completed trajectories into outcome rewards and assigns them back to the action policy. Retrospective Learning retains structured post-hoc hindsight to supervise behavior. Prospective Learning uses hindsight to supervise predictions formed from information available before interaction.

Retrospective Learning from Agent Experience.

Beyond scalarizing a trajectory into an outcome, retrospective methods retain structured hindsight z^{\mathrm{hind}}=\mathcal{R}(x,\tau) extracted by a retrospection \mathcal{R} after interaction—encoding environment feedback, a selected solution, or a reusable skill. All such methods can be unified under a single objective: find a policy πθ that maximizes expected utility when the agent has access to hindsight derived from its own experience,

\max_{\theta}\;\mathbb{E}_{x,\,\tau\sim\pi_{\theta}}\Bigl[\mathbb{E}_{z^{\mathrm{hind}}\sim\mathcal{R}(x,\tau)}\bigl[\mathcal{U}(\pi_{\theta},\,z^{\mathrm{hind}},\,x)\bigr]\Bigr], 

where \mathcal{U} is a method-specific utility that measures how well the policy exploits the hindsight. RLVR is a degenerate instance in which zhind=r⁡(τ) is a scalar and \mathcal{U} reduces to the advantage-weighted log-likelihood. On-policy self-distillation (Zhao et al., 2026; Hübotter et al., 2026) sets zhind to a verified trace and \mathcal{U} to the negative Kullback-Leibler divergence between the hindsight-conditioned teacher and the student:

\mathcal{U}_{\mathrm{OPSD}}(\pi_{\theta},z^{\mathrm{hind}},x)=-\sum_{t=1}^{T}D_{\mathrm{KL}}\!\Bigl(\pi_{\bar{\theta}}(\cdot\mid h_{t},\,z^{\mathrm{hind}})\;\Big\|\;\pi_{\theta}(\cdot\mid h_{t})\Bigr), 

where \pi_{\bar{\theta}} denotes the stop-gradient self-teacher, instantiated as the same model conditioned on privileged hindsight, and πθ the student conditioned only on the interaction history ht. Skill- and memory-based methods (Wang et al., 2026; Lu et al., 2026; Xia et al., 2026) extract reusable abstractions or reflections as zhind and optimize \mathcal{U} as downstream task performance.

Despite their differences, these retrospective methods share a common supervision target: post-hoc information is ultimately used to improve the agent’s reasoning or action policy. We next shift the supervision target from what action the agent takes to what the agent could have anticipated before interaction.

3 Learning Prospectively via Self-Retrospection Distillation

3.1 Prospective Learning

Retrospective learning (Objective 2) uses post-hoc experience zhind to supervise the agent’s reasoning or action policy. We formalize its complementary counterpart: let z\in\mathcal{Z} denote a structured prospection of what the upcoming interaction may unfold. Given a task x and its environment context e, we define the foresight distribution as

p_{\mathrm{fore}}=\pi_{\theta}(\cdot\mid x,e), 

without access to the realized interaction. We suppress the specific prospection instruction from the notation; different instantiations may query different aspects of the upcoming interaction.

After interaction, the realized trajectory provides structured hindsight zhind, as defined in Section 2. Prospective learning utilizes this hindsight as privileged context to define a more informed distribution over the same output space:

p_{\mathrm{hind}}=\pi_{\bar{\theta}}(\cdot\mid x,e,z^{\mathrm{hind}}), 

where \pi_{\bar{\theta}} denotes a hindsight-conditioned teacher policy. Thus, pfore and phind differ in whether post-hoc interaction information is available. Prospective learning optimizes the foresight distribution to anticipate what hindsight reveals:

\mathcal{L}_{\mathrm{pro}}(\theta)=\mathbb{E}_{x\sim\mathcal{D},\;\tau\sim\pi_{\theta}}\Bigl[D\bigl(p_{\mathrm{hind}}\;\|\;p_{\mathrm{fore}}\bigr)\Bigr]. 

Unlike retrospective objectives, \mathcal{L}_{\mathrm{pro}} does not optimize the action distribution πθ(⋅∣ht) directly. It trains the agent to internalize, prior to interaction, the patterns of failure and knowledge that its own traces expose: a target remains informative as long as the agent’s anticipation lags behind what its own execution reveals.

Figure 2: Overview of Self-Retrospection Distillation (SRD). The student policy rolls out trajectories and generates anticipated pitfalls and knowledge foresights via a differently prompted call to the same model. Post-hoc experience from the completed trajectory conditions the stopped-gradient self-teacher, whose token-level distributions supervise the student’s own foresight rollout.

3.2 Self-Retrospection Distillation

We instantiate prospective learning as Self-Retrospection Distillation (SRD) with two potential perspectives of foresight: Knowledge, describing knowledge that may be required during interaction, and Pitfall, describing failures that may be encountered. Both views follow the same distillation procedure and differ only in the prospection instruction used to query the model. Within an on-policy rollout group, successful trajectories provide Knowledge supervision, while failed trajectories provide Pitfall supervision, unifying positive and negative interaction experience as two sources of prospective supervision under the same objective. (Footnote: Detailed hindsight, foresight, pitfall, and knowledge prompts are provided in Appendix B.1.)

Hindsight and foresight construction.

For a task x, let {τi}i=1G denote its on-policy rollout group, with verified rewards {ri}. For each rollout i, we use ci to denote its prospective view—Knowledge when ri=1 and Pitfall when ri=0. Let fi=(τi,y∗,ϵi) denote its privileged post-hoc context, where y∗ refers to gold solution or ground-truth answer and ϵi is the corresponding error annotation when applicable. Conditioning on the same view ci, SRD defines a foresight distribution without access to the completed interaction and a hindsight-conditioned distribution that additionally observes fi:

p_{\mathrm{fore}}^{i}=\pi_{\theta}(\cdot\mid x,e,c^{i}),\qquad p_{\mathrm{hind}}^{i}=\pi_{\bar{\theta}}(\cdot\mid x,e,f^{i},c^{i}). 

Here, ci is shorthand for the corresponding prospection instruction. Thus, reward determines the type of prospective supervision provided by a rollout, while foresight and hindsight within that view differ only in whether privileged post-hoc interaction information is available.

Hindsight-foresight alignment.

For each rollout, the student samples a foresight sequence zfore,i∼pforei. The hindsight teacher evaluates the same student-generated prefix with access to the post-hoc context fi. SRD minimizes token-level distillation loss over the rollout group:

\mathcal{L}_{\mathrm{SRD}}(\theta)=\mathbb{E}_{x,\,\{\tau^{i}\}_{i=1}^{G}\sim\pi_{\theta}}\left[\frac{1}{G}\sum_{i=1}^{G}\sum_{l=1}^{L_{i}}D\!\left(\pi_{\bar{\theta}}\!\left(\cdot\mid x,e,f^{i},z^{\mathrm{fore},i}_{<l}\right)\;\Big\|\;\pi_{\theta}\!\left(\cdot\mid x,e,z^{\mathrm{fore},i}_{<l}\right)\right)\right], 

where D is any distribution divergence measure. A natural and stable choice is the generalized Jensen–Shannon divergence, defined for β∈[0,1] as

\mathrm{JSD}_{\beta}(p_{T}\|p_{S})=\beta\,D_{\mathrm{KL}}(p_{T}\|m)+(1-\beta)\,D_{\mathrm{KL}}(p_{S}\|m),\qquad m=\beta\,p_{T}+(1-\beta)\,p_{S}, 

where p_{T}=\pi_{\bar{\theta}}(\cdot\mid x,e,f,\,z^{\mathrm{fore}}_{<l}) is the self-teacher, instantiated as a stopped-gradient copy of the current policy, and pS=πθ(⋅∣x,e,z<lfore) the student.

4 Empirical Study

4.1 Experiment Setup

Tasks & Benchmarks.

We evaluate SRD across four categories and ten tool-integrated reasoning and agentic benchmarks. i) Math includes AIME 2024, AIME 2026, and AMO-Bench (Art of Problem Solving, 2026; An et al., 2025); ii) Code includes LiveCodeBench-v6 (LCB) and OJBench (Jain et al., 2025; Wang et al., 2025b); iii) Search includes HotpotQA, 2WikiMultiHopQA, and BrowseComp-Plus (Yang et al., 2018; Ho et al., 2020; Zhang et al., 2026a); and iv) Agentic includes ALFWorld and WebShop (Shridhar et al., 2020; Yao et al., 2022). The evaluation includes held-out distribution, format, and interaction-horizon shifts across domains, testing whether gains transfer beyond the interaction conditions seen during training.

Training & Data Construction.

All methods and model scales are trained on identical data with the same prompt ordering. For i) Math, we use DAPO-Math-17K (Yu et al., 2025); for ii) Code, we train on the stdin split of LCB and evaluate on held-out LCB-v6’s functional-python problems; and for iii) Search, we follow the Search-R1 (Jin et al., 2025a) training recipe and retrieval setup (Jin et al., 2025b) with a fixed toolset follows Zhang et al. (2026a). For iv) Agentic tasks, we use the standard training split of ALFWorld and the standard WebShop training partition, evaluating ALFWorld on its OOD split. Math, Code, and Search are jointly trained in a single shuffled mixture, while ALFWorld and WebShop form a second joint training mixture. We train Qwen3.5-4B and Qwen3.5-9B (Team, 2026); full dataset splits, rollout budgets, scaffolds, and optimization hyperparameters are provided in Appendix §C.

Baselines.

We compare against three representative post-training objectives, together with the pretrained Vanilla policy. a) RLVR uses a DAPO-style (Yu et al., 2025) group-relative policy optimization objective (GRPO; (Shao et al., 2024)) over verified outcome rewards. b) Self-distillation: on-policy self-distillation (OPSD) is a baseline following Zhao et al. (2026); Hübotter et al. (2026), where a privileged self-teacher provides token-level supervision to the acting policy; RLSD (Yang et al., 2026) reweighs GRPO with OPSD’s teacher-student divergence. For each baseline we additionally report +SRD results, where the privileged prefix is replaced by our structured hindsight zhind and an auxiliary \mathcal{L}_{\mathrm{SRD}} loss is added with weight λ=0.01. Both families are trained with dynamic sampling under the filter each objective requires, which we detail in Appendix §C.4.

	Math	Code	Search	Agentic
Method	AIME24	AIME26	AMO	Avg	LCB-v6	OJBench	Avg	HotpotQA	2Wiki	BCP	Avg	ALFWorld	WebShop
Qwen3.5-4B-Thinking
Vanilla	24.17	29.58	1.75	18.50	40.48	3.73	22.11	58.12	48.25	20.36	42.24	39.25	75.34
OPSD	62.50	54.17	8.75	41.81	49.40	13.96	31.68	58.00	52.38	17.47	42.62	52.88	85.90
OPSD+SRD	63.33	66.25	14.75	48.11	57.34	17.69	37.52	73.25	63.00	22.29	52.85	66.62	87.46
Improved Perf.	+0.83	+12.08	+6.00	+6.30	+7.94	+3.73	+5.84	+15.25	+10.62	+4.82	+10.23	+13.74	+1.56
GRPO	69.58	54.58	14.50	46.22	55.36	18.34	36.85	71.00	60.62	13.25	48.29	67.75	77.21
GRPO+SRD	76.67	70.83	21.25	56.25	63.89	26.46	45.18	74.25	72.00	22.77	56.34	71.00	83.13
Improved Perf.	+7.09	+16.25	+6.75	+10.03	+8.53	+8.12	+8.33	+3.25	+11.38	+9.52	+8.05	+3.25	+5.92
RLSD	67.92	60.42	13.25	47.20	55.75	18.99	37.37	71.38	61.88	19.40	50.89	62.75	86.89
RLSD+SRD	68.75	66.67	17.50	50.97	57.74	18.02	37.88	71.88	61.88	23.37	52.38	68.50	88.14
Improved Perf.	+0.83	+6.25	+4.25	+3.77	+1.99	-0.97	+0.51	+0.50	0.00	+3.97	+1.49	+5.75	+1.25
Qwen3.5-9B-Thinking
Vanilla	19.58	18.75	0.75	13.03	55.75	2.44	29.10	67.88	59.63	23.98	50.50	48.62	86.82
OPSD	54.17	52.92	12.00	39.70	53.57	20.13	36.85	62.38	55.87	25.54	47.93	73.88	91.76
OPSD+SRD	72.92	77.08	20.75	56.92	64.29	20.62	42.46	78.63	71.38	24.70	58.24	77.13	88.15
Improved Perf.	+18.75	+24.16	+8.75	+17.22	+10.72	+0.49	+5.61	+16.25	+15.51	-0.84	+10.31	+3.25	-3.61
GRPO	76.25	70.42	18.50	55.06	67.26	27.60	47.43	75.38	71.63	17.95	54.99	77.13	78.06
GRPO+SRD	84.17	81.67	28.00	64.61	67.50	31.80	49.65	75.62	71.12	25.54	57.43	78.25	84.18
Improved Perf.	+7.92	+11.25	+9.50	+9.55	+0.24	+4.20	+2.22	+0.24	-0.51	+7.59	+2.44	+1.12	+6.12
RLSD	72.08	67.50	18.75	52.78	60.71	22.24	41.48	73.25	67.88	21.81	54.31	66.25	90.95
RLSD+SRD	77.92	78.33	22.75	59.67	66.67	23.38	45.03	77.38	69.13	23.86	56.79	77.25	91.12
Improved Perf.	+5.84	+10.83	+4.00	+6.89	+5.96	+1.14	+3.55	+4.13	+1.25	+2.05	+2.48	+11.00	+0.17

Table 1: Main results on Math, Code, Search, and Agentic benchmarks, reported as avg​@​8 pass rates in percent. Bold and underline mark the best and second-best result per column within each model scale. Improved Perf. rows give the absolute gain in points that +SRD contributes over its own base algorithm, with a deeper green marking a larger gain.

4.2 Main Results

Table 1 reports performance across all task categories, where every +SRD row distills the pitfall hindsight alone; §5 disentangles the contribution of two perspectives and motivates that choice.

SRD broadly improves post-training objectives and transfers beyond training conditions.

SRD improves category-level performance across GRPO, OPSD, and RLSD at both model scales, with only one regression—WebShop under 9B OPSD (−3.61 pp). The gains are often substantial. For 4B GRPO, adding SRD improves the Math, Code, and Search averages by 10.03, 8.33, and 8.05 pp, respectively, while also improving ALFWorld by 3.25 pp and WebShop by 5.92 pp. The effect extends to other optimization targets: at 9B, OPSD+SRD raises the Math average from 39.70% to 56.92% (+17.22 pp) and Search +10.31 pp. The gains extend to the hybrid objective: RLSD+SRD improves ALFWorld from 66.25% to 77.25% (+11.00 pp). These results indicate that SRD is not tied to a particular base objective, but complements pure RLVR, self-distillation, and their hybrid. The gains also persist when evaluation departs from the training interaction regime. On Code, where training uses the LCB stdin format and evaluation uses functional-format problems, SRD improves the 4B GRPO average by 8.33 pp, including an 8.12 pp gain on OJBench. On the ALFWorld unseen environments, SRD improves every base-objective–scale combination, with gains ranging from 1.12 to 13.74 pp. BrowseComp-Plus further introduces a substantially longer interaction horizon (10× or more tool calls) than the training rollouts; SRD improves five of six settings there, by up to 9.52 pp.

SRD mitigates the instability of pure self-distillation.

Self-distillation without an outcome-grounded signal is the least stable baseline here, and its degradation is concentrated on several tool-integrated benchmarks. At 9B, OPSD alone ends up below the untrained policy — by 5.50 pp on HotpotQA, 3.76 pp on 2Wiki and 2.18 pp on LCB-v6 — i.e. the post-training run has spent its budget degrading the very interface it was supposed to sharpen. The symptom is visible in the scaling behaviour: the Math average of OPSD is inverted across scale, the 9B run landing 2.11 pp below the 4B one (39.70% vs. 41.81%), driven by AIME24 (54.17% vs. 62.50%) and AIME26 (52.92% vs. 54.17%). Adding SRD restores the expected ordering and improves both ends: OPSD+SRD reaches 48.11% at 4B and 56.92% at 9B, an 8.81 pp gap in the right direction, gaining 6.30 and 17.22 pp over the corresponding baseline, and all three Math benchmarks now scale monotonically (AIME24 +9.59, AIME26 +10.83, AMO +6.00 pp from 4B to 9B). The same repair shows on the three columns that had regressed below Vanilla, which now stand at 10.75, 11.75 and 8.54 pp above the untrained policy on HotpotQA, 2Wiki and LCB-v6, respectively. These results suggest that prospective supervision provides a complementary learning signal when direct hindsight-to-behavior distillation becomes unstable.

SRD remains effective as the base policy gets stronger.

The benefit of prospective supervision does not disappear at 9B: the best +SRD result increases from 4B to 9B in every task category, with particularly strong improvements in Math and Agentic tasks. Importantly, we observe that the gain from SRD does not increase monotonically with scale, but continues to provide complementary supervision even as the underlying policy becomes stronger. We examine why this signal persists across capability levels in dual-channel SRD in §5, where reward-uniform rollout groups remain prevalent despite a shift from predominantly all-failure to increasingly all-success groups.

5 Research Questions

RQ1: Which Foresight is more Informative?

Prefix	SRD target	AIME26	AMO	Wall-clock
(s/step)
Qwen3.5-4B
trace	—	53.75	15.50	197
trace	pitfall	57.08	13.50	217
trace	+ knowledge	57.08	15.75	221
hindsight	—	63.75	13.25	258
hindsight	pitfall	67.08	16.75	250
hindsight	+ knowledge	67.50	11.50	267
Qwen3.5-9B
trace	—	65.42	17.50	195
trace	pitfall	67.08	21.25	229
trace	+ knowledge	62.92	19.75	243
hindsight	—	77.50	26.50	262
hindsight	pitfall	84.17	26.25	275
hindsight	+ knowledge	80.42	26.50	279

Table 2: Left: Prefix and channel choice in self-distillation, OPSD. Prefix is what the OPSD teacher sees; SRD target is the distilled perspective; green/red marks a gain/loss over the raw-trace no-SRD row, and Wall-clock is the rollout time per step. Top center: the SRD divergence in each perspective during training, on two different domain-shuffled runs on 4B with RLSD; the band is the local interquartile range of the logged points. Top right: which epistemic tokens and the classes they belong to the SRD policy changes. Bottom center: the GRPO+SRD displacement from the untrained base, decomposed onto GRPO’s own direction: blue is how much of that direction is kept and red is the orthogonal remainder, per split. The dashed line is unity. Bottom right: per-token similarity between each objective’s own update direction and the same objective plus SRD, as a cumulative distribution over held-out positions, each median marked where its curve crosses 0.5. The arrows give how far each policy travelled from the base before and after adding SRD.

SRD can learn from two different perspectives, pitfalls and knowledge, as we shown the promising results in Table 1 with only the pitfalls channel. In this section, we show this choice is more an efficiency motivation than a performance deficit. In Table 2 (Left), we vary the two things that define that target: what the privileged teacher has in self-distillation (OPSD), and which channel \mathcal{L}_{\mathrm{SRD}} distills alongside it. Read against the no-SRD row of each block, SRD helps under both prefixes: distilling pitfalls adds 3.3 and 1.7 pp of AIME26 on the raw-trace prefix at 4B and 9B and 3.3 and 6.7 on the hindsight prefix, and lifts AMO-Bench on 4B with the hindsight prefix by 3.5 pp. The gain concentrates on the hindsight prefix, which solidifies that our hindsight-foresight structure is reliable. However, adding the knowledge channel on top of pitfalls then buys nothing and potentially risks. It is the only setting that ever lands below the baseline: 4 pp drops on AMO-Bench at 4B against the baseline, and 62.92% AIME26 at 9B against 65.42%. Its two wins over pitfall are incremental (67.50 vs. 67.08; 26.50 vs. 26.25), and it is the more expensive of the two in every pair, spending more time in rollout, the phase that is 78–87% of one training step. The training curves in Table 2 (top center) make the same point from the optimization side: the pitfall channel’s divergence falls steadily by 47% and 30% over training on the two runs shown, while the knowledge channel stays flat (∼0.0045 and ∼0.003), so the second perspective is somehow redundant and resists being fitted. Therefore, distilling pitfalls alone is a cheaper and safer target.

RQ2: Which Tokens does SRD Reallocate?

In this section, we measure the effect of the SRD on the policy in the context of token categories. We compare GRPO and OPSD against +SRD in the main results (§4.2), respectively, on 5K policies’ own multi-turn rollouts, scoring only the tokens the model itself emitted.

Figure 3: The same shift token by token, on one rollout per arm: rows are the top-4 likeliest continuations under the policy that owns the rollout.

For every token, we accumulate its expected number of occurrences per rollout under the policy that generated the rollout, and take the difference between the two arms (Figure 2, top right); no checkpoint is ever asked to score another’s text. The shift is systematic in the sense that whole lexical classes move together, and in opposite directions. On validation sets in code, \mathcal{L}_{\mathrm{SRD}} raises connective prose by 8.6 expected tokens per rollout, explicit planning language (we, need, think) by 3.7 and words that name the problem’s structure (key, insight, constraint) by 1.4, while lowering inline mathematical notation by 6.7, punctuation by 4.3, digits by 4.1 and inline variable names (x, n, numbers) by 3.0; tool-call syntax remains unchanged (+0.03). The gains and the losses nearly cancel: summed over every class, the shift is +0.6 tokens per rollout, so SRD is not making the policy talk more; it is moving where the policy spends a fixed budget: out of carrying a calculation inline, into naming the problem and the plan, and it leaves the syntax of calling the interpreter untouched. The same redistribution is visible at a single prefix (Figure 3), where it is concentrated: in these two rollouts under GRPO (top) & GRPO+SRD (bottom) the leading continuation moves two to three times as far as the other four (+1.9 against +0.6 nats above, +1.5 against +0.7 below), so \mathcal{L}_{\mathrm{SRD}} re-weights within the local distribution without adding an offset to all of it. Appendix §E.2 has the full class table and two controls: the shift is not a global sharpening, and it does not depend on whose text is scored.

RQ3: Can SRD Learn from Reward-uninformative Trajectories?

Figure 4: Uniform-reward groups waste most of the rollout at both ends of the capability range: (a) usable share of the budget, (b) overall training group composition.

In GRPO, a prompt contributes a non-zero policy-gradient signal only when its rollout group {τi}i=1G contains reward variation. To isolate this regime, we train on the code domain alone, disable dynamic sampling, and compare the same objective as Yu et al. (2025) with and without SRD across four Qwen3.5 scales (Figure Right). Reward-uniform groups consume a substantial fraction of the sampled budget across capability levels: the usable share of the budget never exceeds 63%, and the discarded share is U-shaped in capability (98.0, 39.1, 37.0, 41.3 points), because as the policy strengthens all-zero reward groups fall monotonically (98.0→13.4) while all-one reward groups rise monotonically (0.0→27.9): a weak policy solves nothing, a strong one solves a lot, and neither leaves any outcome contrast to learn from. Appendix A.3 formalizes this pattern and shows how heterogeneity in prompt difficulty raises the uniform-group rate beyond the homogeneous prediction. Model scaling does not eliminate the blind spot, but it shifts it from failure saturation toward success saturation. Since \mathcal{L}_{\mathrm{SRD}} scores a trajectory against its own hindsight-conditioned prospection rather than against its peers, it needs no within-group reward variance and applies to every group. The 2B case is the limiting one: with 98% of groups uniform, the baseline’s reward is so sparse that it never gets off the ground (1.6% peak training success, 0.0% final), whereas the same budget with SRD reaches 60.6% in training.

RQ4: How does SRD Shape the Prospection Distribution?

To ask where that change points, we measure all four displacements from one common origin, the untrained base: A and S for what GRPO and OPSD training did (A=log⁡pGRPO−log⁡pbase, and likewise S), and E, ES for the same two runs with SRD. None is a component of another, so we can split E into the part along A and the part across it, E=\alpha\hat{A}+E_{\perp} with \hat{A}=A/\lVert A\rVert, in the Fisher metric (Table 2, bottom center). On the held-out code split (626,992 positions, never trained on), α/∥A∥=1.30: the +SRD run travels 30% further along GRPO’s own direction, not back along it, and adds an orthogonal component of comparable size (∥E⟂∥/∥E∥=0.49). Neither this nor its OPSD counterpart averages over cancelling halves: cos⁡(A,E)=+0.81 and cos⁡(S,ES)=+0.98, positive at 77% and 94% of positions individually (Figure 2, bottom right). \mathcal{L}_{\mathrm{SRD}} preserves substantial alignment with the host objective while introducing a non-negligible orthogonal displacement, though the two hosts disagree with each other (cos⁡(A,S)=−0.78). What it changes is scale, in opposite directions: it more than doubles the small RL displacement (0.39→0.86) and shortens the large OPSD one (1.97→0.57) along the same ray (ES≈0.28​S). That asymmetry follows from what the term is — a dense per-token divergence to a target the policy derives from itself. Against a host that supplies sparse credits, it adds movement; against a host already dense self-distillation, it’s a second self-referential target competing for the same capacity, and it behaves like an anchor, the run travelling less far along the direction it was already going. With Table 1, where SRD helps under both objectives, the complementarity is measured: an axis extended, an orthogonal component of comparable size, and a step size pulled toward the host’s own. Appendix §E.4 and §E.5 take up the opposite rescaling and the spread across domains.

RQ5: Does Test-time Foresight Help?

	ALFWorld	WebShop
Qwen3.5-4B-Thinking	Pick	Look	Clean	Heat	Cool	Pick2	Avg	Score
Base objective: RLSD
Base	85.12	50.69	58.59	58.13	56.73	61.46	62.75	86.89
+SRD (Pitfall)	88.69	45.14	72.66	70.00	75.00	53.13	68.50	88.14
+ test-time foresight	86.31	59.03	83.59	66.88	85.58	57.29	73.50	88.27
+SRD (Pitfall+Knowledge)	84.52	30.56	46.09	60.00	70.19	34.38	55.87	82.21
+ test-time foresight	82.14	33.33	44.53	60.63	61.54	42.71	55.62	83.80
Base objective: GRPO
Base	88.69	53.47	64.84	57.50	58.65	83.33	67.75	77.21
+SRD (Pitfall)	92.86	54.86	64.06	60.63	62.50	92.71	71.00	83.13
+ test-time foresight	94.05	45.14	68.75	61.25	62.50	86.46	69.63	83.61
+SRD (Pitfall+Knowledge)	94.05	87.50	67.19	58.13	75.00	64.58	75.38	88.42
+ test-time foresight	93.45	93.75	64.84	57.50	69.23	63.54	75.00	88.37

Table 3: Making foresight explicit at inference, on Qwen3.5-4B-Thinking. Each + test-time foresight row re-evaluates the checkpoint above it with the predicted Knowledge/Pitfall blocks emitted before acting. Benchmarks, splits and protocol are unchanged from §C.5, and Avg is on the same convention as Table 1. Cell shading is each value’s gain over the base of its own objective, on the scale of Table 1; base rows are unshaded. Bold: best per base objective.

This section asks what the prospective objective is worth once training is over: whether the choice of distilled channel changes the agentic result, and whether anything further is gained by making the foresight explicit at inference instead of leaving it implicit. Table 3 varies both, under the evaluation protocol of §C.5. In aggregate the answer is narrow. GRPO is the only base objective where the direction is consistent: all four of its SRD arms land above its base, by 1.88–7.63 pp on ALFWorld and 5.92–11.21 pp on WebShop. Under RLSD the sign flips with the channel: Pitfall is worth +5.75 pp while adding Knowledge costs 6.88 pp against the same base, plausibly because RLSD already carries a self-distillatio...