A Small Transformer Trained in 1.5 Hours Beat Many LLMs on ARC

Mithil Vakde trained a small transformer from scratch in 1.5 hours on a rented RTX 5090. The compute cost came to about 67 cents. The model scored 44% on the ARC-AGI-1 public eval, one point behind TRM's 45% and in the same band that left DeepSeek R1, o3-mini, and Gemini 2.5 Pro behind when the TRM paper ran the comparison (arXiv 2510.04871).

This is the third post in his ARC series. The previous result went viral and drew public scrutiny from researchers including Lucas Beyer, Jeremy Howard, and Rohan Anil. The new one raises the score, cuts the cost, and answers every objection in a long appendix.

What the model does

Each puzzle becomes a sequence of tokenized grids. The transformer trains on those sequences from scratch at test time, using both the train puzzles and the eval puzzle inputs, with the eval labels hidden. Test-time training is not a loophole here. ARC is a meta-learning benchmark built around skill acquisition, and Chollet's original paper frames the whole thing as a measure of how efficiently a system gains new skills (arXiv 1911.01547).

The architecture got a 2026 refresh: 8 layers, SwiGLU, RMSNorm, the NorMuon optimizer, 3D RoPE, and per-task embeddings. Inference augments each test input with color and dihedral permutations, inverts the augmentations, and submits the two most common outputs.

The biggest change is the loss function. Previous versions trained on input and output tokens. This one trains on output tokens only, which makes the approach supervised. Score went from 40% to 44%. Vakde admits he does not fully understand why, and the training loss got worse even as scores improved. His guess is finite model capacity. The ablations back him up on what carries the result: drop the 3D RoPE and the score falls to about 24%, drop the per-task embeddings and it falls to about 24%. Representations, not compute, are doing the work.

The lineage it joins

None of these systems pretrain. They learn from the benchmark's roughly thousand examples:

In about a year, from-scratch systems went from solving a fifth of the eval to nearly half of it, and the cost per result collapsed from research-cluster budgets to spare change on a rented GPU.

Vakde also measured what happens when you strip his advantages. Restricting training data to ARC-1 plus ConceptARC still scores about 40%. A CompressARC-style unsupervised variant drops to about 18%. He thinks 65% is reachable within the current framework, and a union of solved tasks across his own runs already reached 55%. One request to contributors: do not add more training data.

Where the skeptics have a point

The 67-cent figure deserves a careful read. It counts lifetime compute: training from initialization plus inference on all tasks, which works because every task is trained at once. Dividing that total by task count would amortize it unfairly against models charged per task, and Vakde now compares only against TRM, HRM, and CompressARC for exactly this reason. LLM cost figures, on the other side, leave out pretraining entirely, so no single cost axis treats every approach the same.

Parameter claims need the same scrutiny. He argues TRM's famous "7M parameters" leaves out O(100M+) trained embedding weights, and that "7M active weights" would be the honest label. Same caution for HRM.

The score itself stays humble. 44% means 56% of puzzles still fail, and 7% on ARC-2 is early days. OpenAI's o1 reached 75% on ARC-1 through massive post-training (ARC Prize 2024 report), and frontier systems still lead the private leaderboard. This result is about efficiency per training dollar, not about beating GPT-class models across the board. Public eval scores for LLMs are also polluted, since the answers circulate on the internet and end up in training data.

And this is one researcher's unreviewed work. The previous round got picked apart in public, and while the appendix answers the objections convincingly, independent replication is still thin.

Efficiency without the training bill

The pattern matters more to us than the puzzle. Small-model efficiency used to arrive with an invoice: distillation runs, quantization-aware training pipelines, weeks of compute before the first benchmark.

Vakde's result says the invoice is optional on the reasoning side: train from scratch at test time, in 1.5 hours, for 67 cents. Our own work says the same on the storage side. vecq, our training-free vector quantization library, gets its 4-bit and 5-bit codes from Lloyd-Max centroids that are fixed constants. There is no training pass to pay for at all.

In our benchmarks, 4-bit codes hit recall@10 of 0.958 at 5.98x compression, scanning at 0.89 ms per query on ARM, and the current release defaults to 5-bit at 4.79x compression with 0.979 recall@10. The same quantization math scales all the way up to an 87GB model on a CPU. Inside our memory engine, the pipeline reached 98.2% recall against ChromaDB and Mem0 without any learned quantizer. The brute-force scan does give back some speed to HNSW, and why we pick that trade is an architectural choice for edge hardware, not an accident.

What to watch

The interesting shift is the price of admission for credible efficiency results: no pretraining run, one GPU, under two hours, spare change. Chollet's skill-acquisition framing finally has a small-model track with real momentum behind it, and Vakde argues the organizers should go further and ban offline pretraining to make ARC a true sample-efficiency test. His own result is the existence proof that such a rule is survivable.

The author believes 65% is reachable and that hand-written GPU kernels could cut his costs another 10x. Either would have sounded like trolling a year ago.

References

1. Mithil Vakde, "44% on ARC-AGI-1 in 67 cents"
2. "Less is More: Recursive Reasoning with Tiny Networks"
3. "Hierarchical Reasoning Model"
4. "ARC-AGI Without Pretraining"
5. Francois Chollet, "On the Measure of Intelligence"
6. "ARC Prize 2024: Technical Report"

This article is crossposted with blog.codecora.dev as the canonical source.