Twice now I have gone looking for a retrain and found the problem somewhere else.
The first time, a third of my output had collapsed into one template. Before spending anything I counted the pattern in my training data. 5 occurrences in 1,610 examples. Then I counted it in the output. 36%. The cause was a single example I had hardcoded into my own prompt months earlier, and the model was copying it.
The second time, quality had gone flat across the board. The system prompt had grown to 224,833 characters, roughly 56,000 tokens on every call, and ninety percent of it was accumulated rules about how the output should sound. The model was doing exactly what I had asked it to do, which was avoid things.
Neither needed new weights.
That is the pattern I would put in front of anyone about to fine-tune. It is the most expensive way to discover you had a prompt bug. About $30 and a five-hour job per run on a 70B model, and the compute is the small part. The real cost is that you now have two variables moving and no clean way to tell which one broke.
So, three questions.
1. Is this a knowledge problem or an identity problem?
Fine-tuning does not teach a model facts. It shifts a distribution.
If the model needs to know things it currently does not, your product details, your documentation, anything that was true last week and false today, that is retrieval. Training knowledge into weights means retraining every time the knowledge changes, which is a subscription you did not mean to sign.
Fine-tuning is for behaviour that stays put. How it writes. What register it falls back to. What it does when the input is ambiguous.
The test I use: if the right answer changes when the underlying data changes, it belongs in retrieval.
2. Have you stripped the prompt back first?
Most requests to fine-tune are prompt bloat wearing a disguise.
Prompts accrete. Something goes wrong, you add a rule, it improves, and nothing ever gets removed, because removing a rule feels like inviting the bug back. Six months later two hundred lines of prohibitions are competing with two lines describing the actual job, and the model attends across all of it.
Before you price a training run, cut the prompt to the smallest thing that states the task, and measure again. If the output improves, you never had a weights problem.
This is uncomfortable, because it means deleting work you were proud of. Do it anyway. You cannot diagnose a system you have only ever added to.
3. Can you actually produce the data?
This is the question that stops people, and it is where most of the effort goes.
Everyone has data. Almost nobody has a thousand or more curated examples of the exact behaviour they want, cleaned, and is prepared to keep cleaning them.
My last build started from 1,418 labelled examples and kept 884 after cleaning. The final training set reached 1,610 rows once other curated sources were added. Here is what got dropped from that base pool, and none of it was optional.
Rhetorical tics I did not want the model to learn: 107 examples dropped for one construction alone, plus smaller counts for two others. If a pattern sits in your training data at any volume, you are teaching it.
Length and register discipline: 302 dropped for being the wrong length for their category. A model trained on mixed lengths produces mixed lengths.
Opener variety: 45 dropped because more than two examples shared the same opening three words. Without that cap the model finds one opening it likes and reuses it everywhere.
Then there was the part I did not plan for.
83 examples were being deleted on every build
The build had a decontamination step. It removed hand-approved examples from one source file, on the assumption they had already been copied into a curated file elsewhere.
Nothing had ever copied them.
83 examples were dropped silently at build time, on every run, for months. They were disproportionately the best ones, the examples carrying concrete specifics rather than general statements, because those were exactly the ones I had approved by hand in the first place.
A row that never arrives leaves no trace. Nothing in a training run tells you what it did not receive.
So there was no error. The row counts looked plausible. The model trained, the evaluation ran, and the result was slightly worse than it should have been in a way no single test could isolate.
I found it by reading the builder line by line.
That is not really a lesson about checking your pipeline. A training pipeline fails quietly by design. Code that drops rows produces a smaller number, and a smaller number still looks like a number. So print what you dropped and why, broken down by reason, on every single build, and then read it.
The bar
Answer all three and fine-tuning is a reasonable thing to do, and it works. Voice in particular is something you can instruct a model toward but cannot make native to it, and that gap is real.
You will also need an evaluation set that can fail, held out and genuinely separate from anything you trained on, and a rollback path that restores the previous model together with its system prompt. Those are their own articles, and they are the difference between shipping a fine-tune and gambling on one.
But answer the three questions first. Two of them cost nothing except honesty.
