There is a variable called slp — sea-level pressure. Our API documentation lists it as available. Our error messages list it among the valid options. Our fine-tuned model, asked about air pressure over Belgrade, will happily write you fifteen lines of R to fetch it.
The request comes back 400.
It has been coming back 400 the whole time.
What this is
We run dailymeteo.com — a daily meteorological archive for Europe, gridded at 1 km, running from 1961 to roughly five days ago. There's a chat endpoint where you ask a question in plain language and get back R code that queries the archive and answers it. Behind that sits a fine-tuned GPT model, trained on a few hundred question-and-code pairs.
Last week I set out to retrain it on a better dataset. I expected to spend the day on hyperparameters. I spent it finding out that the model had been taught things that were not true.
The training data was built from the documentation
This is the part worth stopping on, because I suspect it is extremely common.
When you build a fine-tuning set for "write code against our API", the natural move is to sit down with the API documentation and write examples from it. That is what had happened. Each example carried a system prompt describing what the API does, a question, and the R code that answers it.
The trouble is that documentation is a claim about a system, not the system. And nobody had checked the claim in a while.
So before touching anything, I did the boring thing: I called the service and wrote down what actually came back.
Three of its claims were wrong.
Claim 1: slp is an available variable
The data exists. There are 21,916 daily rasters sitting on disk, covering 1961 to 2020, Europe only.
The API will not serve any of them. The allow-list in the view is:
VARS = ['tmax', 'tmin', 'tmean', 'prcp']
and the check var not in VARS guards every entry point. On top of that, the continent mapping points at a newer data folder that never received the pressure rasters at all. Two separate reasons for the same 400.
The best detail: the error message you get back still lists slp among the available variables. The message is older than the list.
The model had been taught to ask for it. Two examples in the training set did exactly that. So on any question about pressure, the model produced confident, well-formed, non-functional code.
Claim 2: "data from 1960 to 2024"
Both ends wrong.
1960 returns "There's no data for date range." The archive starts in 1961. And the far end isn't a year at all — the archive is kept near-real-time. Daily data run to about five days ago, monthly to the previous month, annual to the last complete year. Writing a fixed end year into a system prompt guarantees it will be wrong within twelve months, silently.
While measuring this I found something genuinely useful: asking for a range outside the archive is not an error. The API quietly clips.
ann 1901 → 1970 returns 1961 → 1970
ann 1961 → 2035 returns 1961 → 2025
mon 1961-01 → 2030-12 returns through 2026-07
That changes what "correct" code looks like. Generated code doesn't need to know where the archive ends — it can ask wide and read the actual extent back out of the response. That's a pattern that never goes stale. Several of our training examples had been hardcoding an end year instead, which ages badly and quietly.
Claim 3: the timestamp format
The docs said the returned timestamp is formatted like the input date. For aggregated data, true. For long-term means, not remotely:
ann "1961-1990" "1991-2020"
mon "05.1961-1990" "05.1991-2020"
day "25.07.1961-1990" "25.07.1991-2020"
It carries the climate period, not a date. Any code doing substr(timestamp, 1, 4) to pull a year out gets nonsense. Some of ours did.
And the thing that wasn't documented at all
Long-term means — the most semantically awkward corner of the API — had no description whatsoever. The rule, once measured, is simple: the year inside the date selects which climate period you get. Pass 1961, get 1961–1990. Pass 1991, get 1991–2020. Omit the year, get both.
Nobody had written that down. So across 23 calls in the training set, long-term means were invoked seven different ways, two of them mutually contradictory. The model wasn't learning a convention. It was learning that there isn't one.
The subtler poison: examples whose answer the question doesn't determine
This one I didn't expect, and it's the one I'd most like other people to check for.
Consider this pair from the training set:
Question: plotting temperatures during autumn (September 1 to November 30) in Prague
Code:
from = "1999-09-01", to = "1999-11-30"
Why 1999? No reason. The question doesn't say. Whoever wrote the example picked a year.
There were 28 examples like this. And here is why they matter beyond tidiness: the model cannot possibly predict the answer from the question. No amount of training reduces the error on that example, because the target contains information the input doesn't.
You can see it in the metrics. In the previous training run, the loss spikes that survived all the way to the final epoch — steps 223, 238, 249, 253, 260, still spiking at 0.46–0.66 while everything around them sat at 0.13 — were these. They're not a hyperparameter problem. They're irreducible.
Worse, what the model does learn from them is the behaviour: invent a year, say nothing. In production that's a model quietly answering a different question than the one asked.
The fix cost nothing and didn't change a single choice the code makes. We just made it say so:
cat("No period was specified in the question - using 1961 to 2020.\n\n")
Same year. Now the answer is determined by the question, plus a disclosure the model can actually learn.
What we changed
Nine steps, but the shape is simple:
- Measure the API. Every boundary verified by calling the service. The result is one Python file that every other script imports — a single source of truth that is not the documentation.
- Audit against it. A script that walks each example and flags what doesn't work, what wastes model capacity, and what's cosmetic. First run: 13 examples the API rejects, 28 inventing periods, and a training set split 135-to-62 between two different code formatting styles for identical tasks.
- Fix what's unambiguously wrong. The 13 rejects, and 23 silent period choices turned into disclosed ones.
- Normalize style. Half the set broke calls across multiple lines, half kept them on one — same task, double the tokens. That split was costing model capacity on a question with no informational value.
- Fill the gaps. There was not a single example covering questions the API can't answer — wind, humidity, forecasts. The model had been improvising. Now it answers "I don't have that, here's what I do have" — still as runnable R, because the contract is that every answer executes.
- Derive the explanations from the code, not by hand, so they stay in sync when the code changes.
- Split off a validation set. 29 examples the model never sees.
- Preflight. Refuse to upload if anything above regressed.
- Train, then choose.
The previous run had no validation set
This is the single change that mattered most, and it's the cheapest one.
The earlier job ran three epochs with no validation file. Its training loss fell nicely, from 0.38 to 0.16, and everyone was happy.
Training loss falls whether the model is learning or memorising. Without a held-out set, those two are indistinguishable. You are looking at a number that goes down in both the good case and the bad case, and concluding things about it.
With validation, this run:
| Epoch | Training loss | Validation loss |
|---|---|---|
| 1 | 0.368 | 0.285 |
| 2 | 0.133 | 0.131 |
| 3 | 0.071 | 0.140 |
| 4 | 0.038 | 0.152 |
Training loss keeps falling all the way to 0.038 — near-perfect reproduction of what it was shown. Validation bottoms out in epoch 2 and turns back up. Textbook. Invisible without the held-out set.
One incidental finding: the platform's auto hyperparameter selection picked a learning-rate multiplier of 0.51 on the previous run and 2.0 on this one. The only meaningful difference was that our examples got longer. If you rely on auto, know that it can quadruple your learning rate because you added a paragraph to your system prompt.
Then loss and behaviour disagreed
Validation loss says take epoch 2. So I built a second evaluation that asks a different question: does the generated code actually work? Each checkpoint answers the 29 held-out questions, and we measure whether R parses it, whether the API arguments are valid, whether it invents packages that aren't installed, and whether it discloses the period it chose.
| Checkpoint | Validation loss | Failures out of 29 |
|---|---|---|
| Epoch 2 | 0.131 | 4 |
| Epoch 3 | 0.140 | 1 |
| Epoch 4 | 0.152 | 0 |
Exactly inverted.
The explanation is that validation loss measures token-level similarity to a reference answer. It does not measure whether code runs. As training continues, the model drifts away from the reference answer's exact wording — which raises the loss — while the code stays correct and the behaviours we deliberately added keep consolidating.
Epoch 2 has the best number and cannot do the thing we trained it to do: on three of four questions with no stated period, it silently picks a year.
We shipped epoch 4. Had we trusted the loss, we'd have shipped the one that doesn't work.
And then I got the measurement wrong too
Worth including because it's the mistake I'd most likely repeat.
The behavioural evaluation gave epoch 4 a perfect score on disclosure: 4 out of 4. I reported 100%.
It asked each question once.
Re-running the same question against the deployed model several times:
temperature = 1.0 discloses 3/4
temperature = 0 discloses 1/3
The real figure is somewhere around 50–75%, not 100%. The behaviour is genuinely learned and genuinely better than the 25% the previous model managed — but it is not reliable, and a single sample per question cannot tell you that. One sample measures what a model did once. It says nothing about what it does.
(temperature = 0 also isn't deterministic without a seed. Three calls, three different answers.)
What I'd tell anyone fine-tuning against their own API
Your model inherits your documentation's lies. If the training data was written from the docs, every stale claim in them is now a learned behaviour. Call the service and write down what comes back. It took an afternoon and found three errors in a prompt that had been in production for months.
Check whether the question determines the answer. Any example where the target contains information absent from the input is teaching the model to make things up. It also shows up as loss spikes that never come down, which is a cheap way to find them.
Ship a validation split, even a tiny one. Twenty-nine examples were enough to reveal an overfitting turn that was completely invisible for three epochs previously.
Measure whether the output works, not whether it matches. Loss is a proxy. Parse rate, valid arguments, no hallucinated dependencies — those are the thing itself. When the two disagree, the proxy is not the one to trust.
Sample more than once. A percentage from one draw per question is not a measurement, it's an anecdote with a decimal point.
The model was maybe a fifth of the work. The rest was going back and asking the system what it actually does — which, in hindsight, is what anyone should have done before writing the documentation the model learned from.
