A local LLM evaluation should answer a product decision. Producing one more leaderboard number is not enough.
I freeze the target and baseline first, run the unchanged model and candidate through the same protocol, inspect target slices and regressions, and then record one of four outcomes: ship, route, retry, or reject.
Define the contract
The evaluation contract needs inputs, expected outputs, allowed variation, protected content, and failure classes.
Metrics should match the task. SQL needs execution correctness. Tool agents need tool and argument accuracy. Autocorrection needs unnecessary-edit and meaning-preservation checks. Retrieval needs grounded-answer criteria.
Qualitative review still matters, but it should not overwrite deterministic gates.
Freeze the evaluation set
Training, validation, and test data need separate provenance. I deduplicate across splits and freeze the holdout before tuning the recipe.
A holdout that changes for every candidate cannot support a fair comparison. Synthetic fixtures should stay labeled as synthetic, and a small suite should never be described as production proof.
Run baseline and candidate symmetrically
Both models need the same prompts, decoding settings, runtime, and evaluator. Raw predictions and measurement state should be preserved.
Timeouts and missing values are different from zero. When the candidate is a routed specialist, the target slice and the routing boundary both need evaluation.
Measure regressions directly
A specialist can improve its hard gate and become worse elsewhere. I report the delta for every protected slice and set reject conditions before the run.
Those conditions can include meaning changes, leakage, excessive breadth loss, or latency and memory outside the target machine's budget.
Make the decision reproducible
The final package should identify the base and candidate, dataset revision, evaluator revision, configuration, per-slice results, missing measurements, and decision.
"Ship" must remain narrower than "the candidate improved." A routed ship can be correct when the target gain is real and the route is enforced.
PostTrainLLM includes fixture and no-GPU gates, live evaluation commands, canonical factory-run records, packages, and public report cards. Missing historical latency, RAM, throughput, or raw traces remain missing instead of being replaced with estimates.
The full evaluation workflow is at https://posttrainllm.com/evaluate-local-llm.
