LLVM #13 — 30 Runs, Statistical Rigour, and Consistency Checks.

Well hello there! (I know, I know... it's been a minute 😅)

A while back, I ran a quick sanity check on my custom loop-hint pass using a 2D matrix addition and saw the time drop from over a second down to a couple of hundred milliseconds. Which felt fabulous in itself, until I figured that we shouldn't claim a result based on a single manual run alone.

You see, computers are noisy. Background processes can steal cycles, caches warm up unpredictably, and thermal throttling happens. If I wanted to genuinely trust what my compiler pass was doing, I needed to subject it to a rigorous and standardised statistical protocol.

What I built: Initial (Matx Add) Benchmarking Pipeline

What I understood:

1) The Problem with One-Off Benchmarks:

• Running an executable once and quoting the number is fragile. OS scheduling jitter can skew numbers by tens of milliseconds.
• To measure the true performance delta between scalar and vectorised code, we need a distribution over multiple repetitions.

2) The 30-Run Protocol:

• I automated the execution loop to run both binaries (with and without the pass) 30 times back-to-back, collecting the wall-clock durations.

3) Analysing the Variance:

• The standard deviations (7.37 ms for scalar, 10.53 ms for vectorised) turned out to be tiny relative to the overall runtimes.
• This confirmed that the roughly 5.26x speedup (and 81% reduction in runtime) isn't a measurement fluke or random noise. It's consistent, repeatable, and embedded right into what the hardware is doing via SIMD/NEON instructions.
• The gap between our custom hint pass (217 ms) and full -O3 (~140 ms) makes sense: Like I mentioned in previous posts, -O3 unleashes a whole battery of 50+ optimisation passes, whereas our pass targets a single advisory hint (!llvm.loop.vectorize.enable).

What's next: Pushing the hint pass beyond the "happy path" of matrix addition by testing it against reductions (like maybe summing an array), element-wise arithmetic, and strided memory accesses.

Musings:

Lately, I’ve been splitting my time between reviewing these compiler internals and dealing with some compiler-related thoughts at work, alongside tinkering with ideas for another project that's slowly coming together in the background.

I really missed blogging, but for a while, I stopped because I thought there wasn't much point unless it was tied to milestones on an active project. I’ve changed my mind about that because writing things down (project or not) actually incentivises me to keep exploring and learning. So, expect more random thoughts here regardless.

What else have I been up to? I finally clawed my way out of a KDrama slump, started watching 100 Days of Deception, and am currently spending an unreasonable amount of time praying it has a happy ending 🥹

I also started reading Jawaharlal Nehru’s Discovery of India and realised he's a lot chiller than I initially assumed. It was Doordarshan's phenomenal Bharat Ek Khoj, with its insanely good soundtrack, that got me to pick up the book. But even otherwise, I genuinely enjoy reading about India's past. Through all times good and bad, she has been remarkably resilient and steadfast.

In a way, the "idea" of India reflects our own lives: full of unexpected ups and downs, and yet somehow enduring. Having had the chance to experience a few of her very diverse cultures firsthand, I'm always struck by how similar our people actually are underneath, despite their vastly different lifestyles.

No matter where I went, it always felt like home.