On-Policy Self-Distillation for Large Language Models — Detailed Notes
This paper introduces On-Policy Self-Distillation, or OPSD for short, a new way to train large language models to reason better. The core idea is simple but clever: instead of using a separate, larger teacher model to guide a smaller student model, OPSD lets a single model act as both its own teacher and its own student. It does this by giving the model two different contexts during training. In the student role, the model sees only the problem. In the teacher role, the same model sees both the problem and the ground-truth answer or a reference chain-of-thought solution. The teacher version then provides dense, token-by-token guidance to the student version on the student’s own generated responses. This means the model learns from its own mistakes in real time, while having access to the correct answer in the background.
The motivation for this approach comes from watching how human students learn. When a student gets a problem wrong and then looks at the correct solution, they can often rationalize the steps, understand where they went wrong, and internalize the correct approach. The paper hypothesizes that language models have a similar ability: they are better at evaluating and explaining a correct solution than they are at generating the correct answer from scratch. This asymmetry between evaluation and generation is the key insight that OPSD exploits. By giving the model the correct answer and asking it to generate a solution after seeing the reference, the teacher version naturally produces a distribution over tokens that reflects what a well-informed version of the model would say at each step.
Figure 1 illustrates the core OPSD setup: a student policy generates an on-policy response, while a teacher policy (same model, conditioned on the ground-truth solution) evaluates each token. The training objective minimizes per-token divergence between teacher and student distributions.
The technical details of how OPSD works are important to understand. Given a dataset of problems and their ground-truth solutions, OPSD constructs two prompts for the same language model. The student prompt just contains the problem. The teacher prompt contains the problem plus the reference solution and an instruction asking the model to generate its own solution after understanding the reference. Crucially, the teacher does not actually generate tokens during training—it only does a single forward pass to produce a probability distribution over all possible next tokens at each position. The student generates a full response autoregressively. Then for each token position in the student’s response, the training objective minimizes the divergence between the teacher’s distribution and the student’s distribution. The gradient only flows through the student’s logits, so the student learns to match the teacher’s behavior without the teacher itself being updated.
Figure 2 shows the actual prompt templates. The student receives only the problem. The teacher receives the problem plus the ground-truth solution as privileged information, then rationalizes and generates its own solution before evaluating the student’s rollout.
There are three important design choices that the paper explores in detail. The first is which divergence measure to use between the teacher and student distributions. The paper compares forward KL divergence, reverse KL divergence, and the Jensen-Shannon divergence. They find that forward KL divergence works best, because it encourages the student to cover all the modes of the teacher distribution rather than collapsing to a single narrow mode. The second design choice is about generation styles. The Qwen3 model used in experiments supports a thinking mode where it generates self-reflective chain-of-thought tokens. The best configuration turns out to be having the student generate without thinking mode and the teacher evaluate with thinking mode on, because this gives the strongest supervision signal on mathematically meaningful tokens. The third choice is the clipping mechanism. The paper observes that stylistic tokens like “first” or “therefore” can have much higher divergence than math tokens, which would cause the training to focus on style rather than substance. They introduce per-token pointwise KL clipping to prevent this imbalance from dominating the learning signal.
The training process itself is remarkably efficient. OPSD converges within 100 gradient steps, using only a single generation per problem with 1024 tokens maximum. This is in stark contrast to GRPO, a popular reinforcement learning method, which requires 8 generations per problem of up to 16000 tokens each and often sees its performance stagnate or degrade after 100 steps. The reason for this efficiency is that OPSD provides dense, token-level feedback on every single generation, regardless of whether the final answer is correct. GRPO, on the other hand, only gives a binary reward based on whether the final answer matches, and if all eight samples happen to be correct or all incorrect, the gradient vanishes entirely. The paper shows empirically that within 100 training steps, more than half of GRPO’s batches have zero reward standard deviation, meaning no learning occurs at all.
Figure 3 compares OPSD and GRPO on Qwen3-1.7B. OPSD uses significantly fewer tokens but outperforms GRPO on all benchmarks. The rightmost plot shows GRPO’s reward diversity collapse — more than half of its batches have zero reward std dev within 100 steps.
The experimental results are convincing across multiple model scales and benchmarks. The paper tests on Qwen3 models at 1.7 billion, 4 billion, and 8 billion parameter scales, evaluating on three competition-level math benchmarks from the American Invitational Mathematics Examination for 2024 and 2025 and the Harvard-MIT Mathematics Tournament for 2025. OPSD consistently outperforms supervised fine-tuning and matches or exceeds GRPO across all settings. The improvement is most dramatic at the smallest 1.7 billion parameter scale, where OPSD achieves 43.4 percent average accuracy compared to 37.7 percent for GRPO and 35.8 percent for supervised fine-tuning. Even at 8 billion parameters, OPSD achieves 64.8 percent average accuracy versus 64.0 percent for GRPO and 59.8 percent for supervised fine-tuning.
An important finding is that supervised fine-tuning actually degrades performance compared to the base model in many cases. The paper attributes this to the concise reasoning style of the ground-truth solutions in the OpenThoughts dataset. When the model is fine-tuned on these short solutions, it learns to produce shorter answers at test time, which hurts its reasoning ability. OPSD avoids this problem because it does not directly copy the ground-truth solutions. Instead, it uses them as privileged information to guide the teacher, which then provides distribution-level supervision on the student’s own typically longer generations. This distinction between imitating solutions versus learning from them is a subtle but important point.
The paper also compares two ways of computing the distillation objective. The full-vocabulary approach computes the divergence between teacher and student over the entire vocabulary of possible next tokens, which provides richer supervision but is computationally more expensive. The sampled-token alternative computes the divergence only on the token that the student actually generated, using a policy-gradient-style objective. The full-vocabulary approach consistently performs better, suggesting that exposing the student to the complete teacher distribution offers meaningful extra information beyond just the sampled token. There is a clear trade-off involved: full-vocabulary distillation uses more peak memory because it stores vocabulary-sized logits at every token position, but the performance gains are consistent enough to make it the recommended variant.
A notable implementation detail is that the teacher policy is fixed to the initial model weights and is never updated during training. The student policy is the one that gets updated through gradient descent. This design choice stabilizes training and implicitly acts as a regularizer that prevents the student from deviating too far from the initial policy. The paper uses LoRA for parameter-efficient fine-tuning across all experiments, with a rank of 64 and alpha of 128, targeting the query, key, value, output, gate, up, and down projection modules. Training uses the AdamW optimizer with bfloat16 precision and Flash Attention 2 for memory efficiency, running on eight A100 or H100 GPUs.
The paper relates OPSD to a broad landscape of prior work. It distinguishes between hard distillation, where the teacher generates discrete tokens that the student imitates through supervised fine-tuning, and soft distillation, where the student matches the teacher’s full probability distribution. Context distillation and STaR fall into the hard distillation camp, while OPSD is a soft distillation method. STaR is particularly instructive to compare against because it also uses a single model and ground-truth answers. STaR works by having the model generate reasoning traces, filtering out only those that lead to the correct answer, and then fine-tuning on those successful traces. This is essentially a policy-gradient method with a sequence-level binary reward that goes to zero when all samples are incorrect. OPSD, by contrast, provides a token-level reward at every position regardless of whether the final answer is correct, enabling it to learn from mistakes rather than discarding them. Related methods like ReST and bootstrapping approaches follow the same hard-distillation pattern of generate, filter, and fine-tune. The concurrent works SDPO and SDFT explore similar self-distillation ideas but in different settings: SDPO uses environment feedback as privileged information, and SDFT applies on-policy self-distillation to continual learning tasks. In the robotics literature, DAgger is a classic on-policy imitation learning method where a human teacher provides corrective supervision on states visited by the student policy, which shares the same on-policy spirit as OPSD.
The paper also connects to the broader debate about whether supervised fine-tuning or reinforcement learning leads to better generalization. It cites work showing that SFT tends to rely on memorization while RL generalizes better. This is consistent with the paper’s own finding that SFT on concise reasoning solutions degrades performance: the model memorizes the short reasoning style rather than learning to reason robustly. OPSD can be seen as a middle ground that provides the dense feedback of supervised learning with the on-policy distribution of reinforcement learning, without requiring a separate value function or reward model.
An interesting observation from the ablation studies is that earlier tokens in the student’s generation may contribute more to effective distillation than later tokens. When the student’s generation is longer, later tokens become increasingly predictable to the teacher because the teacher has already seen a long prefix of the student’s reasoning. This means less learning signal comes from those later positions. The paper finds that increasing generation length from 1024 to 4096 does not consistently improve performance, supporting the idea that critical branching points in reasoning happen early.
Figure 4 (top) shows that per-token pointwise KL clipping prevents performance collapse on AIME24. Figure 5 (bottom) shows that increasing generation length from 1024 to 4096 does not lead to consistent improvements, as earlier tokens contribute more to effective distillation.
The paper builds on the lineage of DeepSeek-R1 and R1-zero-style training, which showed that reinforcement learning with verifiable rewards can induce reasoning capabilities in language models. OPSD offers an alternative that achieves comparable or better results with far fewer sampled tokens and simpler infrastructure, no need for multiple rollouts per problem or a separate reward model.
What is the research trying to make possible? OPSD is trying to make post-training of language models for reasoning tasks dramatically more efficient and accessible. Currently, the best methods for improving reasoning require either large amounts of high-quality supervised data, expensive reinforcement learning with multiple samples per problem, or a separate larger teacher model. OPSD shows that a single model can teach itself using only the ground-truth answers that are already part of most reasoning datasets, with a fraction of the computational cost. This makes advanced reasoning training feasible for smaller teams and smaller budgets.
What assumption does it quietly depend on? OPSD quietly assumes that the model is already capable enough to rationally evaluate a correct solution and produce meaningful token-level guidance. If the model is too weak to understand the reference solution, the teacher distribution will be no better than random, and the student will learn nothing useful. The paper acknowledges this as a limitation: if reasoning problems exceed the model’s comprehension threshold, the teacher cannot provide meaningful supervision. This means OPSD is not a bootstrapping method for models that start from scratch—it is a refinement method for models that already have some reasoning capability.
What becomes obvious after reading it that was not obvious before? It becomes obvious that the distinction between on-policy and off-policy training is even more important than previously appreciated. The fact that supervised fine-tuning degrades performance while OPSD improves it, using the same underlying data, shows that how you use the data matters as much as what data you use. It also becomes obvious that binary outcome rewards in reinforcement learning are fundamentally wasteful. When a model generates eight different reasoning paths and every single one arrives at the same wrong answer, GRPO learns nothing, but OPSD can still learn from the teacher’s guidance on where each reasoning path went wrong. The paper makes the case that token-level signals are always available even when sequence-level signals are not.
Where does the idea break if you push it outside the paper? OPSD would likely break for tasks where there is no clear ground-truth answer to provide as privileged information. The paper focuses on mathematical reasoning where answers are verifiably correct or incorrect. For open-ended tasks like creative writing, summarization, or dialogue, there is no single correct solution, so the teacher would not have useful privileged information to condition on. The idea also breaks if you scale it to a model that is too small or not sufficiently instruction-tuned, because the teacher would fail at the rationalization step. Additionally, the paper only tests up to 8 billion parameters, so it is unclear whether the benefits persist at frontier model scales. It is also worth noting that OPSD still requires the ground-truth solutions to be available in the training data, which limits its applicability to domains where such solutions are scarce.
What long-running problem did this paper move, even slightly? This paper moves the problem of making language model training fundamentally more sample-efficient. For years, the dominant approaches to improving LLM reasoning have required either massive amounts of data, massive amounts of compute, or massive teacher models. OPSD demonstrates that by rethinking the training signal itself—from sparse binary rewards to dense token-level divergences, and from separate teacher-student setups to self-distillation—we can achieve better results with dramatically fewer resources. It brings together ideas from knowledge distillation, on-policy learning, and self-training into a single coherent framework, and in doing so, it suggests that the field has been leaving useful learning signals on the table. The concrete advances are modest in absolute numbers but significant in terms of the efficiency ratio. More than the numbers, OPSD shifts the conversation from asking how big the teacher needs to be to asking how we can design training objectives that extract maximum signal from every single token the model generates.