TL;DR: most production AI tasks are not LLM tasks. To triage my email, I replaced a 7-billion-parameter model with a tiny classifier in Go. The rule fits in one sentence. Rules first, a small model next, the LLM only as a last resort. The result: no GPU, sub-millisecond inference, and a cloud call that became rare. Here is how, with the real numbers.
This article is for developers who put an LLM in production and pay the bill. Not a demo.
Most AI tasks are not LLM tasks
In 2026, the default reflex is to wire a big model into everything. A question comes in, you call the LLM. But many tasks do not need it. Filing an email under "work" or "newsletter" is classification. A problem solved for twenty years, long before LLMs.
To classify is to pick a label from a short, stable list. To generate text is something else. The first job needs a small model. The second earns a big one. The rule I defend fits in one sentence. Put the LLM last.
The setup
I built an agent that triages my inbox. It is a daemon. It reads new messages and files each one into a category: work, notification, newsletter, promo, and a few more. Nothing secret, just my real mailbox, with years of mail.
The first version handed every email to a local LLM. A 7-billion-parameter model, Qwen 2.5 7B, served by Ollama on a GPU. Ollama is a tool that runs an LLM on your own machine. It worked. But the price was heavy. A GPU on all the time. One more container to watch. And an absurd slowness for the question asked.
One day I asked myself: does deciding "is this a newsletter?" really need 7 billion parameters? No. The answer sits in two or three words from the sender and the subject. So I rethought the whole thing.
Three layers, from cheapest to most expensive
Every email goes through three layers, in order. It stops at the first one that can answer.
- Deterministic rules. Instant, exact, no cost.
- A small model. Sub-millisecond, on CPU.
- The LLM. Only if the small model is unsure.
The routing code fits in a few lines. It tells the whole story.
func (c *Classifier) decide(m Message) Decision {
// 1. Deterministic rules: obvious senders, decided at the door.
if d := preClassify(m); d != nil {
return *d
}
// 2. Small ML model: sub-millisecond, on CPU.
pred := c.ml.Predict(m)
if pred.Confidence >= c.threshold {
return decisionFrom(pred)
}
// 3. LLM last: only the uncertain tail reaches the cloud.
return c.classifyWithLLM(m)
}
The logic is simple. Each layer costs more than the one before. So each layer only handles what the others could not decide.
Rules skim off the easy mail
A large share of my mail is obvious from the sender alone. A known newsletter address is always a newsletter. A platform alert is always a notification. No intelligence required.
A short list of rules decides these at once. It looks at the sender domain and sets the label. Zero guessing, zero model call, zero cost. These emails never reach the small model, let alone the LLM.
Why not let the model do it? Because a rule you can read beats a prediction you cannot, when the answer is obvious. A rule is stable, testable, and free. You keep it for everything that is certain.
A small model for the ambiguous middle
For the rest, the part that is not obvious, I use two old techniques. TF-IDF and logistic regression.
TF-IDF turns text into numbers. Each word gets a weight based on how common or rare it is. Logistic regression is a simple model. It learns to separate categories from those numbers. Together they make a solid, light text classifier.
I train it in Python, with scikit-learn, on nearly 5,800 labelled emails across 6 categories. Then I export it to a plain JSON file. That file weighs 2.4 MB. Compare it to the 7B model: several gigabytes and a GPU.
The key point: inference is 100% Go. No Python, no GPU, no C dependency. The Go code reads the JSON and predicts in well under a millisecond, on a plain CPU. It all fits inside my production image, a distroless image with no shell and no system tools.
And the accuracy? In 5-fold cross-validation, the model reaches 81% correct. Cross-validation splits the data into 5 parts. You train on 4 and test on the 5th, in turn. It is an honest measure, taken on mail never seen during training. The model is strong on the frequent, clear categories, around 0.88. It is weaker on the rare, fuzzy ones, around 0.62.
Finally, the model returns a confidence, between 0 and 1. Above 0.60, I trust it. Below, the email moves to the next layer.
81%, so what?
On its own, 81% looks mediocre. It is not a problem, because of the cascade. No layer has to be perfect. Each layer just has to do what it is good at.
The rules decide the easy mail, without error. The small model handles the bulk of the middle, with confidence. The LLM only sees the tail, the truly ambiguous mail. The expensive call becomes rare. That is the whole point.
You are not chasing a perfect model. You are chasing a system where cost follows difficulty. An obvious email costs nothing. A hard email costs one cloud call. And there are few hard emails.
The token parity trap
There is one tricky part. The model trains in Python but runs in Go. The way you cut text into tokens must be identical on both sides. Byte for byte.
A token is a piece of text the model counts, usually a word. If the Go split differs from the Python split, even slightly, the model sees tokens it never learned. Accuracy drops in silence. No error, just worse predictions.
My fix: a hand-rolled tokenizer, duplicated exactly in both languages. Same regular expression, same table to strip accents, no external unicode library. One explicit table, the same in Python and in Go.
// The same table lives in the Python trainer.
// One divergence and the model sees tokens it never learned.
var fold = map[rune]string{'é': "e", 'è': "e", 'ç': "c" /* full table */}
var wordRE = regexp.MustCompile(`[a-z0-9]{2,}`)
func wordTokens(s string) []string {
var b strings.Builder
for _, r := range strings.ToLower(s) {
if rep, ok := fold[r]; ok {
b.WriteString(rep)
} else {
b.WriteRune(r)
}
}
return wordRE.FindAllString(b.String(), -1)
}
A parity test compares the two tokenizers on real text. If they diverge, the build breaks. A red build beats an accuracy that melts without warning.
The silent regression that taught me a guard-rail
The model improves through a simple loop. When the agent files an email in the wrong place, I move it to the right folder. That move becomes a new label. I retrain, and the model learns from my correction.
One day, this loop bit me. I had re-sorted my mail by hand. One category fell below the minimum number of examples needed to learn it. The retrain dropped it in silence. The new model could no longer predict it. Still no error shown.
The fix: the training script now refuses to drop a category I rely on. It aborts and tells me which folder ran dry.
# Retraining refuses to lose a category in silence.
python train.py --in labeled.jsonl --out model.json \
--expect-labels "work,notification,newsletter,promo,home"
# ABORT: expected classes would be DROPPED from the model: [personal]
The lesson goes beyond email. A silent regression is worse than a crash. A crash, you see. An accuracy that quietly drops, you find out too late. So you make it loud, on purpose.
When the LLM earns its place
I did not delete the LLM. I moved it to where it earns its cost. Writing.
Classifying an email is a closed problem, with few answers. Writing a human reply is an open one. And free language is exactly what a big model does better than anything. So when the agent has to draft a real message, a cloud model handles it.
Small model for the closed task. Big model for the open task. The right tool at each stage. That is the real lesson, not "LLMs are bad". LLMs are excellent. Just not for everything.
The checklist before you reach for an LLM
Before you call a big model on reflex, run the task through these questions.
- [ ] Does the task have a small set of stable answers? Then it is classification, not an LLM
- [ ] Can you write rules for the obvious cases? Do them first, they are free
- [ ] Do you have labelled examples? A small model is probably enough
- [ ] Does the model need to understand free language, or just pick a label?
- [ ] Can you measure honest accuracy, with cross-validation?
- [ ] Does the small model run without a GPU, on CPU, inside your production image?
- [ ] Is your tokenizer identical at training and inference, byte for byte?
- [ ] Does a guard-rail prevent a silent regression on retrain?
- [ ] Do you keep the LLM for what it truly does better, generating language?
What to remember
The 2026 reflex is the big model for everything. Often, the task does not need it. A cascade costs far less and runs far faster. Rules for the obvious. A small model for the middle. The LLM for the only real difficulty.
This is not a rejection of AI. It is engineering. You match cost to difficulty, layer by layer. Building an AI system and watching the bill climb? Want to know where a small model would replace your LLM? That is exactly what I do. Write to me. Keep the big model for what deserves it.
Originally published at jrobineau.com.
I'm Jules Robineau, a senior Go backend and DevSecOps freelancer based in Paris. I build and harden production AI/backend systems at scale (25M+ users). CompTIA PenTest+, Top 1% TryHackMe. Services · GitHub · LinkedIn
Sources: scikit-learn, TfidfVectorizer, scikit-learn, LogisticRegression, scikit-learn, cross-validation
