The puzzle
A teacher judging someone else’s thoughts
On-policy distillation has an appealing story: let the student generate a response, ask a stronger teacher to score every token, and use that dense feedback to move the student toward the teacher.
The subtlety is in the prefix. Each teacher score is conditioned on a trajectory sampled by the student—a sequence the teacher may never have produced. The farther the two policies drift apart, the more the teacher is asked to judge unfamiliar states. A larger teacher is not automatically a more reliable teacher in those states.
At verifiable final-answer tokens, the sign of the teacher’s advantage disagrees with the outcome: a correct answer receives a negative signal, or an incorrect answer receives a positive one.
With a 4B teacher, 20.4% of correct trajectories receive a negative answer-token signal, while 40.8% of incorrect trajectories receive a positive one. With the largest teacher, the signal becomes almost uniformly negative: 97.8% of correct and 96.6% of incorrect answer tokens are pushed down.
Then comes the result that changed the direction of the project. We trained on all trajectories, only those containing noisy signals, and only those without them. The three runs reached comparable performance after a similar number of updates.
If the student improves even on the misleading subset, what exactly is being distilled?
What drives learning
Follow the gradient, not the story
We stripped the objective down along two axes: which sampled tokens actually move the policy, and which signs of the learning signal create the improvement?
Confident tokens are mostly spectators
When the student already assigns a token high probability, its policy gradient is tiny. Teacher and student also tend to agree there, making the advantage nearly zero.
51.7% of tokens have |advantage| < 10−4The negative sign does the work
On the lowest-logp tokens, replacing teacher advantages with the same fixed negative value preserves the improvement. A fixed positive value collapses the policy.
−0.5 works without a teacherThe result is easiest to see at the logit level. For a sampled token with probability p, the update is proportional to A(1 − p). It vanishes when the advantage A is near zero or when the policy is already confident. High-logp tokens often satisfy both conditions.
Δzt ∝ At · (1 − πθ(yt ∣ y<t))
low A → little update · high probability → little updateThis control removes the information that distillation is supposed to transfer. We keep the selected token positions fixed—the lowest-logp 20%—but replace every teacher-derived advantage with one scalar. With A = −0.5, AIME24 performance steadily rises into the same band as standard OPD, while response length grows toward 12K tokens and then stabilizes. No teacher ranking among tokens is needed.
OPD-like gains remain
The selected unlikely tokens are consistently suppressed. Accuracy recovers and the full run remains stable even though teacher logits have been removed.
The policy collapses
Selection is unchanged; only the direction flips. Within roughly 40 steps, response length falls near zero and the gradient norm spikes. Later outputs degenerate into random or garbled text.
The sign ablation isolates the mechanism: what matters is the negative direction on low-probability sampled tokens, not a detailed preference pattern inherited from the teacher.
Working hypothesis
Much of OPD’s gain comes from suppressing unlikely tokens sampled by the student—not from transferring a teacher’s preferred behavior.
The recipe
Turn uncertainty into supervision
A fixed negative signal reveals that the teacher is unnecessary, but it treats every selected token the same. The student already contains a finer signal: its entropy at each position.
Low log probability alone is ambiguous. The model may be uncertain among several good next tokens, or it may be highly confident while a rare tail token happens to be sampled. Entropy separates these cases. We found that stronger negative updates at higher-entropy positions are both more effective and more stable.
Roll out
Sample a response from the current policy and retain token log probabilities and entropies.
Select
Within each response, keep only the 20% of sampled tokens with the lowest log probability.
Self-adapt
Give those tokens negative advantages, increasing the magnitude with normalized entropy.
Aidyn=−12−Hi − Hmin2(Hmax − Hmin)
The conceptual form is shown here; implementations should guard the degenerate case where all selected-token entropies are identical.
This is On-Policy Self-Adaptation, or OPSA. The objective uses the same policy-gradient machinery, but its token-level advantages come entirely from the student’s own distribution. There is no teacher forward pass, no answer label, no verifier reward, and no reference hint. Training still needs prompts; it does not need supervision attached to them.
How it works
Prune the tail, preserve the forks
A negative update does not simply “make the model less confident.” Its effect depends on where the sampled token sits inside the distribution.
Close unlikely branches
When sampling lands in the tail, OPSA pushes that token down and returns its mass to more plausible alternatives.
Share mass across alternatives
When several head tokens compete, lowering the sampled one redistributes mass among the others instead of collapsing to a single path.
Leave certainty alone
High-logp tokens at low-entropy positions are normally outside the selected 20%, preserving precise predictions.
Reasoning often changes direction at high-entropy “fork” tokens—words such as wait, but, or alternatively. OPSA makes the head set more likely as a whole while keeping competition alive inside it. This offers a way to sharpen the distribution globally without erasing the branches that exploration needs.
Evidence
Self-adaptation changes both accuracy and behavior
We trained on the questions from DAPO-17k without their answers, then evaluated across mathematical reasoning, code generation, and general question answering.
+263.5% relative
+264.4% relative
+307.2% relative
| Model | AIME24 | AIME25 | HMMT25 | |||
|---|---|---|---|---|---|---|
| Avg@32 | Pass@32 | Avg@32 | Pass@32 | Avg@32 | Pass@32 | |
| Qwen3-1.7B | ||||||
| Base | 13.44 | 40.00 | 9.69 | 30.00 | 5.73 | 23.33 |
| + OPSA | 48.85 | 80.00 | 35.31 | 66.67 | 23.33 | 50.00 |
| Point gain | +35.41 | +40.00 | +25.62 | +36.67 | +17.60 | +26.67 |
| Qwen3-4B | ||||||
| Base | 23.33 | 56.67 | 20.52 | 56.67 | 13.13 | 33.33 |
| + OPSA | 62.08 | 83.33 | 58.44 | 83.33 | 37.40 | 60.00 |
| Point gain | +38.75 | +26.66 | +37.92 | +26.66 | +24.27 | +26.67 |
| Qwen3.5-9B | ||||||
| Base | 76.35 | 93.33 | 56.04 | 93.33 | 44.48 | 86.67 |
| + OPSA | 87.81 | 96.67 | 76.98 | 96.67 | 67.40 | 93.33 |
| Point gain | +11.46 | +3.34 | +20.94 | +3.34 | +22.92 | +6.66 |
The improvement is not confined to a small base model. OPSA also lifts Qwen3-4B and Qwen3.5-9B, and produces gains on MBPP+ and GPQA-Diamond. Relative to the best compared baseline for each aggregate metric, the 1.7B model gains 11.04 points in mean Avg@32 and 8.89 points in mean Pass@32 across the three math benchmarks.
A behavioral clue
More “wait,” not just more tokens
Response length rises through training, but the change is structured: reflective markers become much more frequent. Masking positions whose head set contains those fork tokens largely removes both the length growth and the accuracy gain.
A diversity check
Diversity holds under our measures
Across 32 responses per problem, pairwise 4-gram diversity approaches that of the base model. Together with the Pass@32 gains, we observe no diversity collapse under these two measures.
The paper in one sentence
OPD can work by suppressing unlikely sampled tokens—not by copying a teacher.
That mechanism motivates OPSA: select the lowest-logp 20% and scale a negative update with the student’s own entropy, enabling self-improvement without teacher logits, rewards, or hints.