Fine-tuning was the easy part: what shipping ASR for a low-resource language taught me

The fine-tune took an hour and a half on a single consumer GPU. Everything that made that hour and a half mean anything took months.

This is the second post in a series about building speech tech for Crimean Tatar, a language with no datasets to speak of. The first one covered synthesis and how the project started; this one is the recogniser — much less a story about training than about measurement.

The headline, so you can decide whether to keep reading:

Left to right: starting model, after fine-tuning, after decode-time tuning. The third bar cost zero training.

Now the parts worth your time.

I gave the model an exam before I tried to teach it

I already had a "working" Crimean Tatar recogniser: a large Whisper, fine-tuned on our language, which I'd been using on clean studio audio where it looked fine. I had read its transcripts and nodded. I had never scored it.

So the first thing I did here was not improve anything. I measured. 34.6% WER — every third word wrong. That's not "rough": it can't back archive search, can't produce training pairs for synthesis, and an assistant built on it mishears you constantly. I only knew that because I finally scored the thing I'd been eyeballing for a year.

One footnote on that baseline: 34.6% slightly understates it, because the model writes 1954 where the book spells out Biñ doquz yüz elli dört and the scorer counts four substitutions. Small, but if I don't say it, the number is dressed up.

Hold out whole recordings, not random clips

Standard practice is a random 90/10 split. Here that would have been meaningless. Adjacent clips come from the same recording, same reader, same microphone, same afternoon — and the segmenter routinely cuts one sentence across two neighbouring clips, so a random split drops half a sentence in train and the other half in eval. What you measure then isn't language ability. It's whether the model memorised a voice and a room.

So I held out two entire books, two readers the model had never heard: 893 clips, 1 h 52 min. One move removes the voice, the mic, the recording era and the vocabulary of a particular text at once. If your data has structure — speaker, session, document, device — split on it. Random splitting is only safe when clips are genuinely independent, and in low-resource audio they essentially never are.

That feels like enough caution. It wasn't.

Audit your eval set by content, not by filename

Before launching anything I checked the obvious thing: is any of my held-out eval material also sitting in training?

By filename: zero overlap. Not one collision. The two name sets looked like they came from different universes.

Then I compared the text instead: for every eval clip, does any training clip contain the same run of six consecutive words? Word 6-grams — cheap to build, cheap to hash, long enough that natural coincidence is rare at this corpus size.

Four audiobooks were in my material twice. Same recordings, segmented by two different tools at two different times, under completely unrelated names.

Four books stored twice. Zero overlap by filename; the collision only showed up in the text.

Top row is the book I was about to grade the model on.

I wasn't panicked. I just understood two things at once: every measurement I'd ever taken on this data was void, and thank God it surfaced before training rather than after.

The counterfactual is the point. If I hadn't checked, nothing would have gone wrong. No crash, no warning, no weird loss curve. The model trains on recordings it will later be examined on — different cuts, same sentences, same voices — posts a great number, and I publish it. The number means nothing: the exam questions were handed out in advance. That's the nasty property of this bug class. It has no symptoms. Everything works, the result is better than expected — which is exactly what should make you suspicious.

And it generalises: in a low-resource language, re-cut copies of the same recordings are the norm, not the exception. There is little material, so you keep returning to it with new tooling, and every pass produces fresh filenames wrapping old content. Deduplicate by content. Filenames are not identity.

After cleaning I re-verified rather than trusting myself: zero eval clips in training, zero 6-gram matches. Measured and written down, not "I'm pretty sure I got them all".

Don't judge a fine-tune by its first checkpoint

Now the part everyone thinks is the project, which was the fastest and dullest bit of it.

I didn't retrain the model. I froze it and trained a small adapter alongside — 2% of parameters, 31M trainable against ~1.5B frozen. 15.5 hours of speech survived the cleanup, 3 epochs, 705 steps, 1 h 30 min on one home GPU, 10.4 GB peak VRAM, and a finished adapter of 126 MB. Result: 20.1% WER, 9.4% CER — a 42% relative cut in word errors.

I felt nothing at the time; a number moved as designed. The pride arrived later, when I ran the model over material it had no business handling — films, podcasts, noisy field recordings — and the transcripts came back near-perfect. That's when it stopped being a table row and became working ears.

Two things from the run are more useful than the headline number.

Checkpoint 100 was worse than the base model. I saved every 100 steps and scored all of them. The first checkpoint came out noticeably worse than what I started from: the adapter has half-learned the new behaviour, it's already interfering with the old one, and it isn't giving anything back yet. If I'd glanced at it and concluded the approach was a dud, I'd have killed a run that ended up cutting error by 42%. A dip at the first checkpoint is what partial adaptation looks like, not what failure looks like.

All the gains landed in epoch 1 of 3. Past roughly step 200 the numbers stopped moving; epochs two and three contributed nothing measurable. So "1.5 hours" is itself padded — half an hour would have done. Checkpoint often and score often; runs converge long before the schedule says so.

One detail that reassured me more than the average: both eval books improved, and the smaller, later, different-reader one improved more. That's the signal you want — the model didn't just learn one narrator's delivery.

Decode-time wins are real — decompose them anyway

Then error dropped again with no training at all.

A trained model has knowledge, but it also has a decoding policy. The default is greedy — take the locally most probable token, never reconsider. Fast, occasionally catastrophic: take one wrong turn and it can't back out, so it goes in circles. That's exactly what the worst eval clips were. Not mishearing — looping:

Qıznıñ qıznıñ qıznıñ qıznıñ …
– dedi Akimoviç, – dedi Akimoviç, – dedi Akimoviç, …

The fix is beam search: hold several candidate sequences and score whole hypotheses rather than committing token by token. Costs decode time, costs zero training. I swept 24 configurations on a separate selection set, not on the exam (more below), then ran the winner on the test set once. 17.0% WER, 7.0% CER. Not one weight changed. Decoding got 3.1× slower.

Here it would have been easy to write "beam search: −3 WER points" and move on. True, and half the truth. So I broke the win down.

409 word errors were recovered in total. 160 of them came from 3 clips out of 893. 0.34% of the material delivered 39% of the gain. Those are the loopers: on 41 reference words the greedy model emitted 175 words of repeated garbage. The remaining 890 clips improved by a modest, even 10%.

Both halves are real, and neither alone justifies a 3.1× slowdown. But they mean different things. The broad 10% is "the model got more careful". The three clips are "it stopped occasionally losing its mind". For archive transcription the second matters far more, because a clip containing 500 bytes of one repeated token isn't merely bad — it poisons anything you train on it later. After the change: zero looping clips, 0 of 893.

Decompose your aggregate wins. If most of your improvement lives in a handful of examples, you didn't improve accuracy; you fixed a failure mode. Those call for different follow-up work, and averaging hides the difference.

The obvious anti-loop knob broke real language

There's a more direct tool against repetition: forbid the model from emitting the same n-gram twice. On paper, exactly the right hammer.

I tested it. It was net harmful: across 255 selection clips it broke 17 and fixed 1, and 12 of the 17 it broke had been essentially perfect. The transcripts explain themselves. The constraint doesn't stop the model repeating — it forces it to repeat incorrectly:

book        – Qartbaba, qartbaba, – tez-tez çapıp kelgen kiçkene Aziz
unrestricted– Qartbaba! Qartbaba! – tez-tez çapıp kelgen kiçkene Aziz
restricted  – Qartbaba! Qartbabaa! – tez-tez çapıp kelgen kiçkene Aziz
harder      – Qartbaba! Qartbava! – tez-tez çapıp kelgen kiçkene Aziz

A child runs up calling "Qartbaba, qartbaba!" — grandpa, grandpa. The model heard it perfectly. The ban wouldn't let it write down what it heard, so it corrupted the second word to avoid matching the first. Elsewhere bağışla… bağışla… ("forgive me… forgive me…") came back the second time as the non-word Bağuşla; in a third case the model routed around the constraint by inventing a suffix, parçalarnıñ parça instead of parçalar. Doubled address and doubled pleading are ordinary Crimean Tatar. People talk that way.

This is my favourite finding of the stage, and it isn't really about decoding. You cannot fix a language with prohibitions. Any rule of the form "this never occurs" eventually collides with how people actually speak, and it breaks precisely that spot while leaving everything else intact — so you won't notice. A decoding default borrowed from an English-language tutorial encodes assumptions about English. Test it against your own language's rhetoric first.

Three sets, and the test set gets decoded once per decision

I've mentioned a "selection set" twice. Every number above rests on it. There are three sets, not two.

• Train — the model learns here.
• Dev / selection — 255 clips, half an hour, two short whole works pulled out of the training material. This is where I choose: which checkpoint, which of the 24 decode configs. Dozens of evaluations, as many as I liked.
• Test — the two held-out books. Decoded exactly twice all stage: once on the winning checkpoint, once on the winning decode config. The selection rule was written down before I looked at any result.

Why so strict? Because if you run the test set repeatedly and take the best result each time, you're no longer selecting the best model — you're selecting the luckiest one. The number comes out pretty and never reproduces.

The dev set got the same cleanliness audit: zero shared clips, zero 6-gram matches. After the duplicate story I don't file that under paranoia.

Your dev set measures ranking, not absolute quality

The dev set was much easier than the test set. Same model: 17% WER on dev, 34.6% on test. Both honestly held out, neither seen in training. But the dev material comes from the same family of recordings as training, while the test books are two entirely foreign readers.

So dev is good for "config A beats config B" and useless for "the model errs at X%". Report the dev number as your result and you're not lying exactly — you're reporting the easier exam. Keep both in their lanes: dev ranks, test reports.

The tool I couldn't validate gets no number

One consequence of that discipline. Whisper ships a temperature-fallback mechanism against repetition: it detects circular output and retries. Precisely the tool this problem wants, and nearly free at decode time.

But my dev set contained zero looping clips. Nothing for it to act on — it looked like a config that does nothing at all. By my own selection rule I couldn't pick it, and I refused to sneak it onto the test set afterwards to see whether it would have helped. So I have no number for it, and I'm not inventing one. If you couldn't validate a tool on the set you're allowed to validate on, write down that it's unvalidated and keep it out of the results table.

17% in the studio. What about the world?

Everything above was measured on one kind of audio: one person, one mic, quiet room, reading a book. Honest for its own world, and it says little about the world a voice assistant lives in. So I took the same 893 clips and corrupted them 17 ways — noise, music, competing speech, reverb, telephone-band filtering, time-stretching. The reference text is known, so error is measurable in each.

• Telephone band and aggressive compression are free. Through a telephone-band filter the model does a hair better than in the studio. Speech rate barely matters either: ±15% tempo costs at most 6%.
• Competing speech is the enemy, by a wide margin. Babble costs roughly 2× what steady noise at the same loudness costs, ~4× what music costs. At equal loudness with the target, WER goes from 17% to 69% — the model stops working.
• Reverb is second. A hallway-sized room multiplies error by 2.5×.

That points somewhere specific: the highest-leverage investment in robustness is not more data. It's source separation and microphone placement. More training hours will not fix a model whose input holds two people at equal volume.

One last check that made me happy: on an open Crimean Tatar audiobook set the model had never trained on, clean chapters landed at essentially the same number as my own exam. It didn't memorise my readers. It hears the language.

What I'd actually tell you

Training was 90 minutes. The months went into auditing the eval set by content, splitting by recording, checkpointing and scoring instead of trusting the schedule, decomposing wins instead of quoting averages, refusing to let an English-shaped default constrain our grammar, and keeping a test set clean enough that its number means something.

Fine-tuning is the part with a progress bar. It is not the part where the work is.

Second post in a series. Coming up: a synthesised voice reading an entire book — sixteen chapters, nearly two hours of audio — and what had to be verified chapter by chapter. Then the one where I rented somebody else's GPUs and set money on fire, computing almost nothing.

Audio samples and project page: https://ai.ana-yurt.dev/.

Two ways to help. If you speak Crimean Tatar, the highest-value contribution is fifteen minutes with headphones — listen to a clip, say whether the machine wrote it down correctly, fix a transcript. No programming needed, and ears like that caught things here that no automated check did; write to support@ana-yurt.com. If you'd rather fuel the work directly, there's ko-fi.com/anayurt.