I cut transfer by 81–257 KiB. Did LCP actually improve? 30 Lighthouse runs

"We made it lighter" and "we made it faster" are different claims, and a single Lighthouse run can't tell them apart — the last time I checked, repeated runs under the same conditions had a spread of more than a second. This is the acceptance sheet I ended up with, and the 30 runs behind it.

Setup

• old = commit 88b702d: stale font subset version, two bold characters missing from the core subset, PNG images, no width/height, regular weight not split.
• new = commit 1ba941a: versioned font URLs (?v=sha1), a build gate for bold characters, WebP, image dimensions, the regular weight split into core/rest, plus three rounds of visual fixes.
• Both built with Astro from git worktrees on the same Windows PC and served with astro preview on :4321 / :4322.
• Lighthouse 13.5.0, --preset=perf --form-factor=mobile --throttling-method=simulate, headless Chrome 153.
• Three pages (home, one Lab article with two figures, the pricing page) × 5 runs × old/new = 30 runs, alternating old→new→new→old→… so time drift doesn't land on one side.
• Hypotheses fixed before running: H1 home LCP goes down. H2 the article's CLS goes down (it got image dimensions). H3 pricing LCP goes down. Stop rule: if a difference is smaller than the 5-run interquartile range, the verdict is HOLD.

const SERVERS = { old: 'http://localhost:4321', new: 'http://localhost:4322' };
const ORDER = ['old', 'new', 'new', 'old', 'old', 'new', 'new', 'old', 'old', 'new'];
// Windows: chrome-launcher fails to delete its temp dir (EPERM) and exits 1 although the JSON is written —
// judge by the file's existence, not the exit code

Numbers

Transfer is bytes ÷ 1,024 (KiB), summed from Lighthouse's network-requests.

• FCP medians: 3,010→2,260 / 2,708→1,959 / 2,709→1,959 ms. TBT 0 everywhere.
• Interquartile ranges (2nd vs 4th of the 5 sorted values — the stop rule uses this, not the min–max above): old 0–145 ms, new 1 ms. Simulated throttling is almost deterministic, so the differences are far outside the noise.
• CLS did not move. Both versions sit at 0.000–0.001. The figures on the article are below the fold, so width/height never had a chance to show up in this metric. H2 is simply not observable this way.
• LCP element is text on all three pages (h1, p.sub). The lcp-breakdown-insight audit puts it on the render-delay side (183–258 ms); what it was waiting for, the breakdown alone doesn't say.
• benchmarkIndex (Lighthouse's host-CPU estimate) — per-page medians old→new: home 2,985→3,246, article 3,239→3,304, pricing 2,193→2,242. Higher on the new side for all three pages. Lighthouse's CPU slowdown is relative to host performance, so this is a bias in new's favour, and I don't attribute the whole difference to the change set. One observation the other way: within old's five runs on the home page, benchmarkIndex moved from 2,417 to 3,329 while LCP stayed within 3,460–3,610.

What went wrong

• First run: chrome-launcher's temp-dir cleanup threw EPERM and the process exited 1 while the report JSON had been written. Fixed the runner to check for the file.
• Lighthouse 13's perf preset no longer has largest-contentful-paint-element; the phases live in lcp-breakdown-insight.
• The frozen "new" commit and the measured one differ. The design froze 098df1a; HEAD moved four commits (four Lab articles added) before I ran. The diff touches none of the three pages' HTML or CSS; the font woff2 files grew by a total of 2.6 KiB (ten bold characters added) — under 1% of transfer. I did not re-run; this line is the deviation record.

Verdict, and what the verdict means

• H1 supported (−600 ms, outside the range). H3 supported (−454 ms). H2 not supported (old was already 0).
• "PASS" here means: the pre-registered stop rule did not fire, the difference was observable. It does not mean the change set has been proven to cause the speed-up — the CPU index leaned toward new, and WebP, fonts and dimensions were changed together, so no per-change attribution is possible.
• Not measured: field data. Simulated throttling estimates other conditions from one observed trace; whether real users got faster is a separate question for Web Analytics / Search Console / CrUX after enough days.

The three-column sheet

Put "bytes went down" and "LCP went down" in separate columns and sign off on each. Building both versions on one machine and alternating five runs each was enough here to separate the observed difference from the run-to-run spread; it is not enough to say what caused what.

Part 1 of the series "Measure the Outcome, Not the Proxy" (part 2: shorter ≠ requirements preserved; part 3: screenshot success ≠ threshold discrimination — both on this account).

Verification record (environment, verdict, the 30-run CSV, the LCP breakdown CSV and the runner) and the canonical Japanese write-up. What changed between old and new: the font-subset story and the image/metadata generators.