One Integer Deleted the Stop Token From My Loss. The Curve Never Noticed.

For a long time I had a model that trained cleanly and produced nothing usable. The loss fell. Gradients were finite. Nothing crashed. It simply never learned to stop — every generation ran to the length cap and got truncated.

The cause was two lines of my own source that were individually correct.

The collision

In a neural codec language model the audio vocabulary has a fixed size, and the stop token is one extra class on top. So the output layer is one wider than the codebook:

nn.Linear(d_model, NUM_AUDIO_TOKENS + 1)   # 1025 classes: 0..1023 audio, 1024 = EOS
eos_id = NUM_AUDIO_TOKENS                  # 1024

Correct. EOS is the last class and the layer has room for it.

Then, elsewhere, the loss:

F.cross_entropy(logits, targets, ignore_index=NUM_AUDIO_TOKENS)

Also reasonable on its own. ignore_index is how you skip padding.

But the sentinel and the stop token are the same integer. Every position whose target was "stop" was discarded before the loss was computed. Not down-weighted — removed. The model was never once shown an example of stopping, across every epoch it ever ran.

PyTorch defaults ignore_index to -100 precisely because it must be a value that can never be a real class. Replace it with a real vocabulary constant and that guarantee is gone, silently: shapes valid, loss finite, run healthy.

The curves are identical

Minimal reproduction, two arms differing only in the sentinel value.

The broken arm finished at 0.0035. The fixed arm finished at 0.0034.

Same curve to any human, any dashboard, any threshold you would write. One has a working objective and one has an objective with a hole in it, and the loss cannot distinguish them — because the loss is computed over what survived the mask. A metric cannot report on examples it never received.

The second half: lowest loss, worse model

200 epochs at lr = 2e-5, evaluated every 50 on a fixed split held out by utterance (n = 32). Rank is the stop token's position among 1025 classes at the true terminal frame.

Between epochs 100 and 150 the training loss improved by 20% while mean P(stop) fell 54%, top-1 stop accuracy went 18/32 to 8/32, and autonomous termination halved.

Selecting by lowest training loss returns epoch 150. The model that terminates reliably is epoch 100. The reversal was observed independently in a shorter run, which is why I am willing to state it.

"Save the checkpoint with the lowest validation loss" is the default in more or less every training script in existence, including mine. On this run it was actively the wrong rule, and the number it optimised looked better the whole way down.

For completeness, what fixing the collision bought on the real model, same held-out split:

End-to-end synthesis then terminated on its own at frame 203 against a 350-frame ceiling. Before the fix that was impossible by two independent mechanisms: the training-time collision above, and an inference-time mask that set the logit of every index at or beyond the codebook size — including end-of-sequence — to negative infinity before sampling.

One honest loose end: P(stop) plateaus in the range 0.35–0.47 across two learning rates and a 5.9x increase in training data (224 to 1313 utterances, with speaker, language, emotion and the held-out split all held constant). So the plateau is not a data-quantity limit. I have no confirmed explanation for it. Terminal timing in speech is genuinely ambiguous, and hedging with the stop token ranked second of 1025 may simply be correct behaviour.

The same integer, safe one stage over

The detail I find most instructive: the identical line is harmless in the next stage of the same model.

The autoregressive stage predicts the first codebook plus EOS — 1025 classes, so 1024 is a real class and using it as a sentinel is fatal. The non-autoregressive stage predicts audio codes only — 1024 classes, so 1024 is out of range and the exact same ignore_index is correct.

One + 1 in a different file decides whether that line destroys your objective. Both stages read identically at the call site. That is not a mistake you catch by reading carefully; it is a mistake you catch by checking a relationship between two numbers that never appear together.

Which is what a linter is for

This is now two rules in trainproof, my linter for training runs:

• Sentinel collision. Compare the output layer's class count against ignore_index. If the sentinel is a valid class, fail. If it sits exactly one past the end, say so explicitly — because the same integer is fatal one class earlier, and that distinction deserves to be visible rather than silently passed.
• Dead class. Accumulate which classes ever reach the loss as a positive target during the first epoch, then flag any class the output layer can emit but that never once appears as an answer.

My own two stages are the regression fixture — the fatal case and its safe twin, one integer apart. Not a synthetic example.

The design decision worth stating: the dead-class rule only fires when coverage is already broad and few classes are missing. One unseen class out of 1025 is a structural exclusion. Nine hundred unseen is a small sample. Without that guard the check screams on every short run and gets switched off — which is how good checks die.

If you take one thing from this

I do not think this is rare. Any codebase where a padding sentinel, an end-of-sequence id and a vocabulary size are all defined as named constants in different files can produce it, and none of your instrumentation will complain.

If you fine-tune anything with a custom ignore_index, go and check it against your output layer's width right now. It takes thirty seconds and the failure mode is completely silent.

Paper: The Loss Curve Is Not a Sufficient Statistic — Silent Objective Failures from Sentinel-Class Collisions in Neural Codec Language Models

DOI: 10.5281/zenodo.21864658

pip install trainproof — GitHub, MIT