Intro
Every team that plugs an LLM into its business hits the same moment. Someone asks the model about an internal pricing rule, a product SKU, or a clause in the standard contract, and it answers confidently and wrongly.
The reflex that follows is almost universal: "The model doesn't know our business. Let's fine-tune it."
So the team spends weeks exporting tickets and docs, cleaning them, formatting them into JSONL, and paying for training runs. The new model sounds more like the company. It uses the right jargon. And it still makes things up.
That's not bad luck. It's a category error.
What fine-tuning is actually good at
Fine-tuning adjusts a model's weights so it behaves differently. That's powerful when the thing you want to change is behavior:
- A specific tone or writing style (support replies that sound like your brand, not like a chatbot)
- A strict output format (always return this JSON schema, always produce this report layout)
- A narrow, repetitive task where a smaller tuned model can replace a bigger general one
Notice what all of those have in common. They're about how the model answers, not what facts it knows.
When the goal is "the model should know our refund policy, our product catalog, and last quarter's changes," you're asking weights to act like a database. They're a bad one.
Failure mode 1: it still hallucinates, just in your accent
A fine-tuned model doesn't gain a reliable sense of what it knows and what it doesn't. Training on your documents shifts probabilities toward your vocabulary, but when a question lands in a gap, the model does what it always does: it produces the most plausible-sounding continuation.
Illustrative scenario: a support assistant tuned on two years of tickets is asked about a warranty extension introduced last month. It has never seen it. It answers anyway, in perfect company voice, with the terms of the old warranty. The answer is more convincing than a generic model's would have been, which makes it more dangerous, not less.
Fluency in your domain is not the same as accuracy about your domain.
Failure mode 2: every fact change is a training run
Business knowledge isn't static. Prices change, policies get revised, products get deprecated, a regulation lands and three documents get rewritten.
If that knowledge lives in the weights, every update means rebuilding the dataset, retraining, re-evaluating, and redeploying. In practice, teams don't do that weekly. So the model drifts out of date, quietly, and nobody knows exactly which facts are stale.
Compare that with a retrieval setup: a document changes, you re-index it, and the next query sees the new version. The update cycle is minutes, not a project.
Failure mode 3: you can't show your work
When an answer comes from retrieval, you can point to the exact chunk of the exact document it was grounded in. You can show it to the user. You can log it. When someone disputes an answer, you can check whether the source was wrong or the model misread it.
When an answer comes from fine-tuned weights, there is no source. The knowledge is smeared across billions of parameters. You can't cite it, you can't audit it, and you can't tell a compliance team where a claim came from.
For anything customer-facing, regulated, or contractual, that alone should end the discussion.
What to build first instead
Before training custom weights, invest in the boring part: retrieval.
user question
-> query rewriting / expansion
-> hybrid search (keyword + embeddings) over a clean, deduplicated index
-> reranking
-> top-k chunks + source metadata into the prompt
-> answer with citations
Most of the quality comes from things that have nothing to do with the model:
- Clean source documents (no three conflicting versions of the same policy)
- Sensible chunking that keeps related context together
- Metadata (dates, owners, product, region) you can filter on
- Hybrid search, because embeddings alone miss exact identifiers like SKUs and error codes
- An evaluation set of real questions with known correct answers, so you can measure whether changes help
None of this is glamorous. All of it pays off more than a training run for knowledge-heavy use cases.
When fine-tuning does make sense
This isn't "never fine-tune." It's "fine-tune for the right reason, and usually later":
- Retrieval is solid, answers are grounded, but the output format or tone keeps drifting. Tune for behavior.
- You need a smaller, cheaper model to handle a narrow, high-volume task a big model already does well. Tune for cost.
- The model struggles to use retrieved context well in your domain (for example, dense technical or legal text). Tune on examples of reading and citing context, not on the facts themselves.
In all three cases, facts still come from the index. The weights handle behavior.
The reframe
Fine-tuning teaches a model how to talk. Retrieval tells it what's true right now. Most "the model doesn't understand our business" problems are the second kind wearing the costume of the first.
What pushed your team toward fine-tuning (or away from it), and did it actually fix the knowledge problem you were trying to solve?
