A GPU Is Two Things, and Only One of Them Holds Your Model

I have run AWS infrastructure for years. When I started looking at how large language models actually get served, I hit a wall on something basic.

Someone said "the model sits on the GPU." Someone else said "the GPU does the calculation." Both sounded right. Neither told me where the model actually lives.

The problem is that "GPU" is not one thing.

Two parts, one board

You already know how a normal machine is laid out. A CPU does the work. RAM holds the data. They sit apart, connected by a bus.

A GPU card is the same pair, shrunk onto one board:

• Cores — the chip in the middle. Thousands of small processors that do maths.
• VRAM — memory chips around it. Holds the data.

So when people say different things, they are talking about different halves:

Neither half is "the GPU" on its own.

The cores hold nothing

This is the part that took me longest to accept.

The cores have almost no storage. They are hands, not shelves. The model does not get loaded into them once and stay there. It has to be streamed across from VRAM, continuously, for every calculation.

An 8 billion parameter model at full precision is about 16GB of numbers. Those numbers live in VRAM. To produce a single word of an answer, the GPU streams the whole thing across to the cores, does the maths, and gets one word back. Then it does it again for the next word.

That one fact explains why answers type themselves out instead of appearing all at once.

Where this bites you on AWS

Here is the practical part.

Look at a GPU instance in the EC2 pricing console:

g6.4xlarge    $1.3232/hr    16 vCPU    64 GiB    1 x 600 GB NVMe

That 64 GiB looks like plenty of room for a model. It is not GPU memory. It is the server's ordinary system RAM, and inference barely touches it.

The GPU on that instance is a single NVIDIA L4 with 24GB of VRAM. That 24GB is the only number that decides whether your model loads.

The EC2 pricing page has no GPU column at all. You have to cross-reference the instance-types page to find the card and its memory.

Worth memorising for the common G-family instances:

One sentence to take away

Weights live in VRAM. Cores do the maths. Neither one is "the GPU" by itself.

Everything else in model serving follows from that split.