Quality Filtering
We add our quality filters incrementally. Without any filtering, OPST already improves over the base model by 5.2 points. Pseudo-verification adds 2.7 points, and the consensus gate a further 2.3 points.
On-policy self-distillation enables post-training foundation models without relying on stronger external teachers. However, existing approaches rely on ground-truth question-answer pairs, human-curated problem sets, or other supervised signals. In this work, we reformulate on-policy self-distillation as a self-training framework, driven by self-generated question-answer pairs and privileged reasoning demonstrations. We push supervision to the limit: guided solely by coarse, high-level topic descriptions, the model synthesizes its own curriculum of queries and privileged reasoning traces without access to any human-curated prompts or answers. We find that self-distillation on fully synthetic privileged information enables a model to improve across multiple domains while retaining its capabilities on domains it is not self-trained on. We demonstrate the generality of this framework across text and multimodal domains, achieving consistent self-improvement on standard language benchmarks and Visual Question Answering. Our findings demonstrate that foundation models possess rich latent capabilities that can be amplified without external task supervision or curated problem distributions.
Post-training usually relies on human-written queries and reference answers, or at least on a curated prompt distribution. In our data-free setting, neither is available: training is seeded only by a set of coarse topic descriptors, such as "Abstract Algebra" or "Commonsense Causal Reasoning". Given a topic, the model poses a new problem and, along with it, generates privileged information consisting of a rationale and a candidate answer. The generator is either a frozen copy of the initial model or the EMA teacher, so that the quality of the generated data can improve as training progresses.
Removing human curation also removes quality assurance, so generated samples can contain ill-posed questions or incorrect answers. We filter them along two axes that act in opposite directions. Pseudo-verification checks that a sample is well-formed: the model must recover the candidate answer when solving the question in a fresh context, and again after paraphrasing the question, which rejects problems that admit more than one reading. The consensus gate then checks that a sample is informative: if every on-policy rollout of the student already reproduces the answer, the teacher has little to add and the sample is discarded. The retained problems therefore lie at the student's capability frontier, and since the gate depends on the current student, the curriculum becomes harder as the model improves.
On the filtered samples, we train the model with on-policy self-distillation. Student and teacher share the same model, but only the teacher is conditioned on the privileged information, while the student sees the question alone. The student samples its own rollout, and the teacher supplies a next-token distribution at every step, which the student matches under a bounded, skewed Jensen-Shannon divergence. Crucially, the generated answer is context for the teacher rather than an optimization target: the teacher produces its supervision from its own parametric knowledge, filling in reasoning steps the rationale omits and limiting the influence of errors in the generated data. The teacher is an exponential moving average (EMA) of the student, which stabilizes training against self-induced drift and is also the model we evaluate.
Generation, verification, and training are repeated over multiple rounds, with student and teacher weights carried over between rounds and duplicate problems removed. Student and teacher co-evolve: a stronger teacher provides more accurate token-level targets, generates better problems and privileged information, and judges its own data more reliably as the verifier.
OPST is modality-agnostic. For vision-language models, we start from a collection of unlabeled images and visual topic descriptors such as "Spatial Relations", "Object Counting", or "Attribute Recognition". The model poses questions about each image, and the privileged information is either a detailed scene caption written by the model, spatial relations bootstrapped from its own object detections, or both. Visual verification requires the candidate answer to hold across augmented views of the image, so that answers resting on hallucinated premises are filtered out.
We self-train the Gemma 4 E2B and E4B models with topic descriptors covering the domains of MMLU, GPQA, GSM8K, and HellaSwag, and evaluate with and without thinking. To measure transfer and retention, we additionally evaluate on ARC-Challenge, MATH-500, and Winogrande, which no topic descriptor targets.
We compare training algorithms on identical self-generated data: SFT imitates the generated answer, GRPO rewards agreement with it, and OPST supplies it as teacher context. On Gemma 4 E2B without thinking, OPST improves MMLU by +10.2, GPQA by +4.0, and HellaSwag by +7.5 points. On average, OPST exceeds SFT on the same data by 5.6 / 1.3 points (E2B / E4B) without thinking and by 2.7 / 3.8 points with thinking. Since the two differ only in whether the generated answer is the target or the teacher's context, this gap isolates the effect of that choice.
We further compare OPST against methods that use human-curated data. Among all methods without reference solutions, OPST attains the best average in all four settings, exceeding u-OPSD, which trains on curated questions, by 1.0 to 2.8 points and R-Zero, which also self-generates its problems, by 1.4 to 5.2 points. OPST also closes most of the gap to full supervision: without thinking on E2B, it recovers 88% of the gain of supervised OPSD, and with thinking, it matches supervised OPSD at both model sizes.
OPST can also self-train on several domains at once. Combining the topics of all three in-domain benchmarks improves the in-domain average by 7.6 points, within 1.0 point of single-domain training, indicating little interference between domains. Out-of-domain performance is largely retained, with an average gain of 0.8 points, although MATH-500 regresses in this mixture. Combining MMLU with GSM8K topics improves MMLU by 11.3 points, more than MMLU topics alone, and yields the best out-of-domain average (+3.9), suggesting positive transfer from mathematical reasoning.
Visual OPST transfers our framework to vision-language models using unlabeled images from OpenImages alone. On Gemma 4 E2B, both forms of privileged information improve VQAv2 accuracy, and combining scene captions with spatial relations yields the largest gain of +6.9 points. On the POPE object hallucination benchmark, the improvement comes mainly from recall, with gains of +4.5 points at E2B and +6.2 points at E4B.
We add our quality filters incrementally. Without any filtering, OPST already improves over the base model by 5.2 points. Pseudo-verification adds 2.7 points, and the consensus gate a further 2.3 points.
Without privileged information, or with a placebo taken from a different problem on the same topic, the gain shrinks to about 3 points. The rationale alone recovers 8.3 points and the answer alone 7.1 points, so the gain stems from sample-specific privileged information.
We disable both filters and replace a fraction p of the candidate answers with wrong ones. At p = 75%, OPST loses only 1.9 points and stays above the base model, whereas SFT loses 5.7 points and falls below it. Conditioning the teacher on the answer, rather than imitating it, limits the influence of incorrect generations.
@article{sick2026opst,
author = {Sick, Leon and Hoyer, Lukas and Str{\"u}mpler, Yannick and Reisswig, Christian and Ropinski, Timo and Tombari, Federico},
title = {On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models},
journal = {arXiv preprint arXiv:2610.32770},
year = {2026},
}