I trained a 142M language model from scratch to play chess. You can play it right now

A few weeks ago I trained a language model from scratch to play chess. Not fine-tuned, not prompted. A plain transformer decoder in PyTorch, 20 layers of width 768, trained on nothing but sequences of moves, on a single RTX 3090.

It plays rated games on Lichess and it will take a challenge from anyone:
lichess.org/@/philidor-142M

Around 1685 in rapid and bullet, across 950+ rated games, and not a single illegal move in any of them.

One move, one token

The design decision everything else follows from: there is no BPE, no subword vocabulary.

There are exactly 1971 tokens. Three specials (<pad>, <bos>, <eos>) and 1968 legal move shapes in UCI notation, every geometrically possible from-square to to-square pair, promotions included.

So a game is not a string. It is a sequence of tokens, one per half-move, and a 200-move game is a 200-token sequence. That is why the model answers in ~23 ms regardless of how long the game has been running, while a general model has to re-read a growing text prompt.

The legality mask is one line

The model, left alone, occasionally proposes an impossible move. In free generation it is legal 98.85% of the time. Good, and disqualifying: a UCI engine that returns one illegal move loses the game on the spot.

The fix is a single masked softmax over the legal moves in the current position:

mask = torch.zeros(self.vocab_size, dtype=torch.bool, device=self.device)
for move in board.legal_moves:
    idx = self.stoi.get(move.uci())
    if idx is not None:
        mask[idx] = True

logits = logits.masked_fill(~mask, float("-inf"))

One forward pass, deterministic, and it preserves the model's ordering among legal moves. That is the whole engine.

Filtering mattered more than architecture

I read 89,288,421 games from the Lichess archives and kept 11,035,777. A 12.4% retention rate.

Everything else follows from that: 790M move tokens, and about 3.2 billion tokens seen during training. A model learns the distribution you show it, and most of what is available is fast games between weak players.

Two traps that had nothing to do with chess

The lazy generator that was not lazy.

Parsing 89 million games in parallel looked like this, and it looks correct:

games = (parse(g) for g in dump)        # a generator, nothing computed yet

with mp.Pool(10) as pool:
    for result in pool.imap_unordered(work, games):
        ...

Pool.imap drains its input as fast as it can, queueing every task up front. On 89 million games that is about 100 GB of text held in RAM, on a machine that has 31. The generator's laziness is completely cancelled by an eager consumer downstream.

The fix is to hand it fixed-size windows instead of the whole stream. Measured footprint after the change: 2.8 GB, flat from start to finish.

The general form of this bug: a careful component loses all its care as soon as an impatient one plugs into it. The part is not at fault, the assembly is, and that is exactly what you cannot see by rereading your own code.

The train/validation split that leaks in silence.

The obvious way to split a token stream:

cut = int(0.99 * len(tokens))
train, val = tokens[:cut], tokens[cut:]   # looks reasonable

It cuts a document in half. The model saw the beginning during training, and you are now asking it to predict the rest as if it were new. Your validation loss improves, your curves look great, and there is no visible symptom at all.

I split by whole game instead: 10,925,420 games for training, 110,357 for validation, appearing nowhere else.

It is the same leak as a time-based split that lets the future through, or a random split that separates two rows belonging to the same user. It never shows up in a metric. It has to be reasoned about.