Originally published on AI Tech Connect.
What you need to know They do different jobs. Continued pre-training runs next-token prediction over raw text at large volume — typically billions of tokens — and updates broad model knowledge. Supervised fine-tuning runs on input-output pairs or preference data at much smaller volume — thousands to millions of examples — and updates behaviour on specific tasks. The plateau is the signal. If doubling a clean supervised set stops moving your metric, you have a distribution problem rather than a data-volume problem, and more examples will not close it. Diagnose before you spend. Tokeniser fertility on your domain text, against a general-English reference, is the cheapest evidence that the base model never really saw your domain. An afternoon, no GPU-hours. Adapt, then instruct. Continued…
