A training method called Co-RL improves a language model's reasoning with no correct answers anywhere in the loop, by rewarding each model for matching a different model's majority vote. In the authors' tests it matches or beats the identical recipe trained with real ground-truth labels in 11 of 14 settings, and it stays stable in exactly the conditions where a model grading its own work collapses.
Key facts
- Co-RL matches or surpasses ground-truth-supervised training in 11 of 14 settings, and matches or beats a leading label-free baseline in 13 of 14.
- The agents share no parameters and exchange no gradients -- only votes.
- Posted to arXiv August 18, 2026 and revised August 19; the paper is arXiv 2608.17253.
- Primary source: the Co-RL project page, with code on GitHub.
The bottleneck this attacks is real and getting worse. Reinforcement learning with verifiable rewards is the technique behind most of the last two years' progress on reasoning, and it requires a checkable right answer for every training problem. As the authors put it, that supervision "is costly to obtain and becomes increasingly scarce as target reasoning capabilities approach or surpass what humans can reliably evaluate." You cannot label your way to a model better at math than the people writing the labels.
The obvious workaround -- let the model reward itself for its own most confident answer -- fails in a specific and well-documented way. The project page states the problem in one line: "Training solely on self-generated feedback can reinforce existing biases and lead to training collapse." A model that is confidently wrong gets rewarded for being confidently wrong, and the error compounds until the run diverges. The paper's training curves show self-rewarding runs leaving the plotted range entirely at multiple scales while Co-RL stays flat.
Co-RL's fix rests on a simple observation: "Independently trained models make different errors. Each one can therefore correct mistakes the other cannot see in its own generations." Each agent samples several answers to an unlabeled prompt and reduces them to a majority vote. A completion earns a reward of 1 when its answer matches the peer's vote, never its own. Then each policy is updated separately with GRPO. Beyond two agents, the votes pass around a directed ring, so no agent ever contributes to its own supervision target.
The analogy is two students grading each other's homework without an answer key. If they studied from the same book and made the same mistakes, the exercise teaches nothing -- they will confirm each other's errors. If they studied differently, each catches things the other missed, and the agreements are far more likely to be right than either student alone.
That analogy also identifies exactly what the method consumes, and the paper is unusually clear about it. The project page reports that different-family pairs start with lower error overlap on math problems than same-family pairs or same-model-different-seed pairs, and that rewriting the training data for one agent decouples them further while preserving the answers. Error diversity is the fuel. When it runs out -- two agents that make identical mistakes -- there is nothing left to learn, and the method degenerates toward self-rewarding.
The headline comparison is against GT-Reward, which the paper describes as the same recipe with the same models, data, and budget, plus the ground truth. Co-RL matches or surpasses it in 11 of 14 settings and matches or beats the label-free baseline TTRL in 13 of 14. Under one multi-agent protocol it reaches a higher score than a competing method while using half as many agents and no separate judge model.
Why it matters: if peer disagreement can substitute for correct answers, the ceiling on training reasoning models stops being the supply of verified problems and starts being the diversity of available models -- a resource that is currently abundant and growing.
The honest caveats: this is benchmark-bound work, trained on a math problem set and evaluated on math and multimodal reasoning suites, and apart from one benchmark every number is a single run evaluated with a single sample. Whether the effect survives on domains where answers are not cleanly checkable is untested. Public reception is still thin -- the repository shows a couple of dozen stars and the paper page carries little discussion beyond the authors. And a majority vote is not truth: if two independently trained models are wrong in the same direction, agreement rewards the shared error just as confidently as it rewards a correct one. It is a close cousin of the failure where a frozen model can look like it taught itself.
Originally published on Ground Truth, where every claim is checked against the primary source.
