Sources this series builds on: Chili's (ChiliTomatoNoodle) multithreading
playlist,
CMU 15-213 (Computer Systems: A Programmer's Perspective), various CppCon talks,
and LLMs for restructuring and sanity-checking. Everything here is my own code,
compiled and run — not just transcribed.
Recap — the question this series has been avoiding
Part 1 left a number sitting there unresolved: the mutex-protected version of
complexFunction took 13.2 seconds — over 4x slower than running the
same work single-threaded with no threading at all. Part 2 showed a plain
bool racing across threads and getting hung or skipped entirely
depending on the optimization level. Both articles explained the C++-level
mechanics — read-modify-write, register caching, the compiler's license to
assume no data race exists — but neither one actually answered why a lock
costs that much time or why one core doesn't just immediately see another
core's write.
Those answers aren't in the C++ standard. They're in the hardware underneath
it. This article is where the series stops treating "memory" as a black box
and starts building the vocabulary to actually answer both open questions —
starting from the most basic thing there is: what a variable even is, once
you strip away the syntax.
A computer, stripped down to two things talking to each other
Underneath everything else, a running program is really just two components
in constant conversation:
- The CPU — does the thinking. It can only perform very small, very simple operations: add two numbers, compare two numbers, move a number from one place to another. Everything complex your code does is millions of these tiny steps happening extremely fast, one after another.
- Memory (RAM) — holds data while the program runs. The CPU itself can only hold a tiny amount of data internally at any given moment (registers, covered below), so anything larger — your variables, arrays, objects — lives out in memory, and the CPU has to go fetch it when it's needed and send it back when it's done.
That fetch-and-send-back relationship is the single most important thing to
carry forward from this article. Almost everything about performance and
multithreading correctness in this series comes back to "how fast can the
CPU get data out of memory, and what happens when more than one core wants
the same piece of memory at the same time."
Cycles — the CPU's metronome
A CPU doesn't work continuously; it advances on a beat. Each tick of that
beat is a clock cycle. On every tick, the CPU can move its work forward
by roughly one small step (a simplification, but the right mental model for
now).
Clock speed — the number on a spec sheet, like "3.5 GHz" — is just how many
of these ticks happen per second. 3.5 GHz means 3.5 billion ticks a second.
So when a later article says "this operation takes 200 cycles," that's not
an abstract unit — it's literally "the CPU sat through 200 of these ticks
before it could continue." At billions of ticks a second that's still a
tiny slice of real time, but it's 200 times longer than something that only
takes 1 tick, and that ratio is what actually matters, not the absolute
time.
Keep this number in your back pocket: reading from a CPU register takes
roughly 1 cycle. Reading from main RAM takes roughly 200 cycles.
That gap — the CPU sitting idle for 200 ticks of its own clock, doing
nothing but waiting for data — is the entire reason caches exist, and caches
are where this hardware track goes next.
Bytes and addresses — a variable is just a name for a location
Memory isn't one undivided blob. It's organized as a long row of small,
individually numbered slots:
- A byte is the smallest individually addressable unit of memory — 8
bits. Most things you care about (an
int, achar) are a small number of bytes glued together. - An address is the position number of a byte in memory — a house number on a very long street. Address 0 is the first byte, address 1 is the next, and so on.
Here's the reframe worth sitting with: int x = 42; is not, underneath
everything, a special protected thing called x. It's a label your compiler
uses for "start at this address, and the value there is 4 bytes long."
Reading x at runtime means the CPU is told an address and goes and fetches
whatever bytes are stored starting there.
This is the seed of everything coming later in the hardware track: two
completely unrelated variables in your program can end up sitting at
neighboring addresses, purely by how the compiler laid them out — with
consequences neither you nor the compiler intended. That's the exact setup
for false sharing, several articles from now.
Fast vs. slow — the ordering that everything else hangs on
You don't need exact nanosecond numbers yet — just the relative ordering,
because that ordering is the skeleton the rest of the hardware track hangs
real numbers on:
Each step to the right is slower to access but holds more data. That
trade-off — and exactly why the hierarchy is shaped like a pyramid instead of
one giant fast memory — is Part 5.
Cores — why "more than one CPU" changes everything
Modern CPUs usually contain more than one independent execution unit, each
able to run instructions on its own — a core. A 4-core CPU can run four
genuinely simultaneous streams of work, not just fast-switching between one
stream at a time. This is what makes Part 1's speedup real: each thread
lands on its own core, actually running in parallel.
But multiple cores introduce a genuinely new problem that doesn't exist with
a single core: each core typically has its own small private cache. So what
happens when two cores both want to read or write the same piece of memory
at the same time? That question — not anything about instructions or
compilers — is the real, hardware-level reason Part 1's mutex cost 13.2
seconds, and it's exactly what the next few parts of this series are for.
Takeaways
- A CPU and RAM are, underneath everything, just two things constantly fetching from and sending data to each other — and that relationship is where every later performance question in this series comes from.
- 1 cycle for a register, ~200 cycles for RAM isn't a rounding error — it's the CPU sitting idle 200x longer, and it's the entire reason caches exist.
- A variable is not a protected, special thing — it's just a name for "some bytes at some address," which means two unrelated variables can end up neighbors in memory without either programmer intending it.
- Multiple cores make CPU-bound work genuinely parallel (Part 1's real speedup) but introduce a brand-new question single-core programs never had to answer: what happens when two cores want the same memory at once?
