Your Local LLM CSV Needs a Schema Version

My RTX 5090 benchmark notebook has measured data I cannot compare without hand repair. The models are not the problem. The CSV files disagree about what a benchmark row is.

The short answer: I audited seven local LLM benchmark CSVs on 2026-08-28. They contain 17 rows across four header widths, three power-field shapes, and zero schema_version fields. A benchmark file needs a versioned row contract before it needs another model run.

Canonical URL: https://bmdpat.com/blog/local-llm-benchmark-csv-schema-version-2026

What did the CSV audit find?

I read every CSV under my 5090 benchmark folder and compared the headers before I compared the results.

Seven files hold 17 data rows. Their headers have 9, 10, 15, or 17 columns. None has a schema_version field. None has a run_id field. Two files have captured_at; the other five do not.

Power has three shapes. Two files store watts_before and watts_after. Four files declare watts_avg, although one row leaves it blank. One file has no power column and puts 158W inside a notes sentence.

Those rows are still real measurements. The problem is that the reader must guess how each file maps into a common table.

Why does header drift break a benchmark notebook?

A CSV reader can join files only after someone decides which columns mean the same thing. watts_after is not the same measurement as watts_avg. A date is not the same as a timestamp. A runtime name is not a run identity.

Unit drift is the sharpest case. My 2026-06-14 and 2026-07-09 files record total_duration_s, eval_duration_s, and load_duration_s. My 2026-08-02 v2 and 2026-08-04 files record total_duration_ms, eval_duration_ms, and load_duration_ms. Same measurement, different unit, different name. The other three files carry no duration column at all. A reader that matches on the word duration and ignores the suffix is wrong by a factor of 1000.

Token rate splits the same way. Six files write tokens_per_second. One writes tok_per_sec. Those names map safely because both mean tokens per second. The values still do not compare, because that lone row is a five-task eval average and the others are single runs. A safe name mapping is not a safe comparison.

My daily 5090 report flattens all 17 rows into one display table. That makes the report readable, but it cannot recover fields that the collector never wrote. A blank power cell may mean not measured, not supported by that schema, or lost in free text.

I found the same boundary when I audited missing driver versions. The row must carry its own facts at capture time. A later report cannot add them without inventing history.

What belongs in a versioned row contract?

I now treat the CSV header as an API. Version one has four groups.

Identity fields name schema_version, run_id, and captured_at. Environment fields name the GPU, driver, runtime, runtime version, model, and quant. Workload fields record the task, input tokens, output tokens, context size, and stop reason. Result fields record wall time, load time, prompt time, generation time, token rate, VRAM, and power.

Each field needs one unit in its name or contract. load_duration_ms is clear. load_duration is not. watts_avg must mean an average sampled during the measured window, not the reading after the request ended.

The contract also needs explicit missing values. I use not_measured when the collector skipped a metric and not_supported when the runtime cannot expose it. An empty string is not enough because it hides the reason.

How should old benchmark rows move forward?

Do not rewrite old files to look current. Keep the original bytes and add a read-time adapter for each known header.

The adapter can map tokens_per_second and tok_per_sec into one output field because both source names remain visible in the mapping. It must not map watts_after into watts_avg. When no safe mapping exists, the normalized value stays unknown.

That preserves the measured result and the limits around it. It also lets a future collector write only the current schema while the report reader handles older files.

This is where power data needs a defined sample window. A schema name is useful only when its measurement rule is fixed too.

What should fail before a new row is saved?

The collector should reject a row with no schema version, run ID, capture timestamp, runtime version, or GPU name. It should reject numeric fields that carry units in the value instead of the contract. It should also reject unknown columns unless the schema version declares them.

Then test both sides. A complete current row should pass. A row copied from an old file should fail with the missing fields named in the error. That negative case matters because a permissive writer is how the notebook reached four header widths.

The fix is small. The collector owns one row contract. The report reader owns explicit adapters for old contracts. No human edits a benchmark row after the run.

Accompanying prompt

What the prompt does: It audits local LLM benchmark files for schema drift and writes a safe normalization plan without filling missing facts.

Copy/paste this prompt:

Role:
You are a schema auditor for local LLM benchmark files.

Context:
I will give you CSV headers and sample rows from several benchmark runs.

Task:
1. Group files by exact header shape.
2. Find fields that use different names, units, or measurement windows.
3. Separate safe mappings from mappings that would change meaning.
4. Draft one versioned row contract for new runs.

Output:
- A table of files, header widths, and schema groups.
- Safe field mappings with their source names preserved.
- Fields that must stay unknown.
- Required fields and fail rules for the new collector.

Constraints:
- Do not infer a missing value from a later system reading.
- Do not map point-in-time power to average power.
- Do not replace the original files.
- Use explicit units and missing-value reasons.

Copy the block above.

Get the next measured local AI build note from The 5090 Reports.

Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=local-llm-benchmark-csv-schema-version-2026

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.