An asymmetric test‑time objective lets on‑policy distillation hit supervised OPSD performance without any hand‑labeled data. By rewarding rollouts that agree with a teacher and penalizing those that diverge, the method turns unlabeled interaction streams into a reliable supervisory signal. The result is label‑free test‑time training that reaches the same accuracy as pipelines built on costly annotation.
Before this work, reinforcement‑learning fine‑tuning and On‑Policy Self‑Distillation (OPSD) both required ground‑truth labels to steer the model toward correct reasoning paths. A natural workaround—using majority‑vote pseudo‑labels—proved brittle: a single incorrect vote could corrupt the teacher and mislead every token in subsequent generations.
Even when the pseudo‑label is wrong, 79 % of the rollouts that disagree with it are wrong too, confirming an asymmetric error pattern that can be exploited safely. This observation underpins the design of TTPO’s dual‑branch loss, where agreement triggers distillation and disagreement triggers a penalizing RL signal[1].
Without any labels, TTPO matches label‑supervised OPSD on five competition‑level benchmarks and lifts Qwen3‑1.7B from 38.0 % to 45.2 % in test‑time training, delivering a +25.2 % to +36.4 % gain when the model is evaluated without additional thinking time[1].
TTPO exceeds the label‑dependent OPSD across all three model scales—40.1 vs. 39.7 on the 1.7B variant, 58.6 vs. 58.4 on 4B, and 62.6 vs. 61.7 on 8B in average score—despite operating without ground‑truth supervision[1].
The approach still leans on majority‑vote pseudo‑labels as a routing signal; if the vote is consistently noisy in a new domain, the asymmetry may weaken and the penalty branch could misfire. Moreover, the experiments focus on mathematical reasoning tasks, leaving open whether the same gains translate to language generation or classification settings.
If these results hold broadly, the standard practice of building a labeled OPSD dataset before test‑time training should be abandoned for math‑heavy pipelines; simply run TTPO at inference time and cut the entire annotation stage out.
