What Every AI Engineer (Builder) Needs to Know About the Data Behind AI

Table of Contents

Chapter 1 of 4, deep dive. Click any line to jump straight to it:

• Introduction
• Module 1.1: The Big Picture: What a Model Actually Learns
  • 1.1.1 The One Sentence Everything Builds On
  • 1.1.2 The Pipeline Loop as Three Mathematical Operations
  • 1.1.3 How Big Is "Big"?
  • 1.1.4 The Quantity, Quality, Diversity Trade-off, Measured
• Module 1.2: The Mathematics of the Training Objective
  • 1.2.1 Entropy, Cross-Entropy and KL Divergence
  • 1.2.2 Training Is Maximum Likelihood
  • 1.2.3 The Cost of Spam, Derived
  • 1.2.4 Proof in Code: Clean vs Spam-Polluted Training
  • 1.2.5 The ln(V) Sanity Check
• Module 1.3: What the Model Is Truly Optimizing For
  • 1.3.1 From Tokens to Loss to Perplexity
  • 1.3.2 How the Data Pushes on the Model: the Gradient
  • 1.3.3 Scaling Laws, With Real Numbers
  • 1.3.4 Compute-Optimal vs Over-Trained
  • 1.3.5 Where the Numbers From the Intro Come Back
• Chapter 1 Cheat Sheet
• References

Chapters 2, 3 and 4 (cleaning the web, teaching filters to think, and the final mile into tokens and batches) publish as their own posts.

Introduction

I have been quiet for a while, not because I had nothing to say, but because I kept turning one question over in my head: how do people like us, without a fraction of the money the big labs spend, actually contribute to building AI that reflects who we are. The honest answer took time to arrive at, and the number that finally pushed me to write this down is too large to sit with quietly.

Did you know? Meta's own published model card for Llama 3.1 states that training the 405-billion-parameter version alone used 30.84 million GPU-hours on H100 chips drawing 700 watts each. Run that math and it comes out to roughly 21.6 million kilowatt-hours of electricity, enough to power about 2,000 average households for a full year. Meta's own figures put the location-based emissions from that one training run at 8,930 tons of CO2, close to what 1,900 cars emit in a year. Spread across the 16,384 GPUs used, that workload still took about 78 days of continuous training, and at a modest $2 per GPU-hour cloud rate, the compute bill alone comes to roughly $62 million, before a single researcher, dataset, or building is paid for.

Sit with that $62 million for a second, then sit with this. In 2021, OpenAI contracted a company called Sama to have Kenyan workers label disturbing content so ChatGPT would not reproduce it, paying those workers roughly $1.50 to $2 an hour while internal documents showed OpenAI itself was billed $12.50 an hour for that same work. Separately, Scale AI, the company Alexandr Wang led as CEO until Meta bought 49 percent of it for $14.3 billion in 2025, has faced its own wave of US lawsuits after reports that its Remotasks platform paid some workers as little as a few cents per task. Two different companies, two separate scandals, the same pattern underneath: the humans doing the most psychologically demanding part of building this technology are paid the least, while the compute bill for the model they helped clean runs into the tens of millions.

Who this is for. This series is written for builders, students, self-taught engineers, researchers and curriculum designers anywhere who want to understand how large language models are actually built, not just how to call an API, especially those of us working with limited compute, limited funding and limited institutional backing. You do not need a GPU cluster or a research lab to read this. You need curiosity, a laptop, and the willingness to look closely at data most people scroll past.

And while that gap sits unresolved, the people steering this industry are moving at a different altitude entirely. In September 2025, Donald Trump told the United Nations General Assembly that the United States would start calling artificial intelligence "superintelligence," or SI, later signing an executive order requiring federal agencies to use the new term. That is the altitude at which this race is being run, nations rewriting policy and vocabulary around who leads next, while much of the data that trains these systems, our languages, our conversations, our culture, keeps flowing outward for free.

This is where the series begins, not with despair at the size of the gap, but with the one part of it we can actually close ourselves. Over the next four chapters we move from the big picture of why training data matters so completely, through where real data comes from and how it gets cleaned, into the filtering and quality techniques that separate a frontier lab's dataset from a pile of scraped pages, and finally into how that data becomes the tokens and batches a model actually trains on. None of it requires a data center. All of it requires showing up.

How to read this post. Every section follows the same rhythm: the idea in plain words, the math (each equation is followed by a "read it as" sentence), a worked example with real arithmetic you can check with a calculator, and why it matters for data. All code lives in one repository so this post stays readable:

💻 Companion repo: https://github.com/iws3/before-we-train. Every script in chapter-01/ runs on a laptop CPU with plain Python and NumPy. No GPU needed.

Notation used throughout:

Module 1.1: The Big Picture: What a Model Actually Learns

1.1.1 The One Sentence Everything Builds On

The idea. A language model becomes whatever you feed it. Not a feeling and not a simplification: it is what the training procedure is mathematically built to do. Whatever pattern sits in the text, grammar, facts, boilerplate, spam, a stereotype repeated often enough, gets absorbed. The model cannot tell your intentions apart from the data's contents. It only ever sees the data.

The math. A language model assigns a probability to any piece of text by splitting it into tokens and using the chain rule of probability:

P(x₁, x₂, …, x_T)  =  P(x₁) · P(x₂ | x₁) · P(x₃ | x₁, x₂) · … · P(x_T | x₁, …, x_T−1)

  or, written compactly:

P(x₁, …, x_T)  =  ∏ P(xₜ | x₁, …, xₜ₋₁)        (multiply over t = 1 … T)

Read it as: the probability of a whole text is the probability of the first token, times the probability of the second given the first, times the probability of the third given the first two, and so on.

For example, the sentence "the cat sat" is scored as:

P("the cat sat")  =  P(the) × P(cat | the) × P(sat | the cat)

A model with parameters θ tries to make its own probabilities match the real ones. Training picks the θ that makes the training text as probable as possible:

θ*  =  the θ that maximizes  Σ log P_θ(documentᵢ)        (add over i = 1 … n)

Read it as: "find the settings θ that give the highest total log-probability to all n training documents." (We add logs instead of multiplying probabilities; 1.3.1 explains why.)

Notice what is missing from that formula: any mention of what is good. The only input is the data. Change the data and θ* changes with it.

Worked example: a model you can compute by hand. The simplest language model is a bigram counter. It predicts the next word using only the previous word:

              n(c, w)
P(w | c)  =  ─────────
               n(c)

Read it as: the chance of word w after context word c is how often c was followed by w, divided by how often c appeared at all.

Train it on three sentences: "the cat sat", "the cat ran", "the dog sat". The word "the" starts all three, so n(the) = 3. After "the" comes "cat" twice and "dog" once:

• P(cat | the) = 2/3
• P(dog | the) = 1/3

Now an attacker, or just a careless crawler, adds three copies of the spam sentence "the free prize". The word "the" now starts six sentences, and "free" follows it three times:

Nobody told the model to like spam. Spam's share of the counts simply became spam's share of the probability. A transformer is far more powerful than a counter and generalizes beyond exact counts, but its training objective (Module 1.2) pulls it toward the same destination: the model's beliefs are the frequencies in your data.

Why it matters. This is why data preparation is not a chore before the real work of building AI. It is the real work. Most open models use nearly the same transformer design. What separates a strong model from a weak one is almost always what it was fed: how clean, how diverse, how deliberately chosen.

💻 Code: chapter-01/1_1_1_bigram_counter.py. Builds the bigram model above, prints the probability table before and after spam, and lets you change the number of spam copies to watch P(free | the) climb toward 1.

Expected Output:

1.1.2 The Pipeline Loop as Three Mathematical Operations

flowchart TD
    A["Sources: web crawl, books, code, research, community data"] --> B["Extraction: raw pages to clean text"]
    B --> C["Filtering: remove junk, duplicates, unsafe content"]
    C --> D["Mixing: decide how much of each source to use"]
    D --> E["Tokenizing and batching"]
    E --> F["Training: the model learns to predict the next token"]
    F -->|"evaluate, then adjust a filter or a weight"| B

Nobody gets the mixture right on the first try. Every strong dataset in use today is the record of a team running this loop dozens of times: train a small model, check what improved, adjust a filter or a weight, repeat. Anyone with a laptop can run that loop at small scale.

The idea. Every box in the diagram does one of three things to the data distribution. Once you see them as operations on a probability distribution, the whole pipeline becomes easy to reason about.

Operation 1: Mixing

If you draw documents from several sources, where source k has distribution p_k and weight w_k, the combined distribution is:

p_mix(x)  =  Σₖ wₖ · pₖ(x)        where  Σₖ wₖ = 1

Read it as: the mixed dataset is a weighted average of its sources, and the weights add up to 100%.

The Llama 3 paper reports a final mix of roughly 50% general knowledge, 25% mathematical and reasoning data, 17% code and 8% multilingual text. Those percentages are the w_k values (they add up to 100%). They are design decisions, not facts of nature.

Operation 2: Filtering

A filter assigns each document a keep-score f(x), which is 1 or 0 for a hard filter (keep or drop) and anywhere in between for a soft one. The distribution after filtering is:

                 p(x) · f(x)
p_filtered(x) =  ───────────        where  Z = Σₓ p(x) · f(x)
                      Z

Read it as: keep each document in proportion to its score, then divide by Z so the probabilities add up to 1 again. Z has a plain meaning: it is the keep rate, the fraction of the data that survives.

Operation 3: Deduplication

If a document appears m(x) times, its weight in the dataset is proportional to m(x). A boilerplate footer that appears in 1,000 of 1,000,000 pages carries 0.1% of the probability, while a unique page carries 0.0001%. The footer is 1,000 times over-represented. Deduplication sets every m(x) back to 1 and removes that distortion.

Worked example 1: filters multiply

Suppose a crawl passes through four stages with these keep rates (illustrative numbers):

The total keep rate is the product:

Z_total  =  0.90 × 0.60 × 0.50 × 0.50  =  0.135

Only 13.5% survives. Planning backwards: if you want 1 billion clean tokens, you must start with

1 billion
─────────  ≈  7.4 billion raw tokens
  0.135

Worked example 2: tokens seen versus tokens owned

A source with weight w_k in a run of D total tokens contributes w_k · D tokens. If your unique data for that source is U_k tokens, the model sees it for this many epochs (passes over the data):

               wₖ · D
epochsₖ  =  ──────────
                Uₖ

Take a 15 trillion token run and a language that gets 0.01% of the mix. Step by step:

1. Tokens drawn: 15 × 10^12 × 0.0001 = 1.5 billion
2. Unique tokens the community has collected: 200 million
3. Epochs: 1.5 billion ÷ 200 million = 7.5

Work on data-constrained scaling (Muennighoff et al., 2023) found that repeating data for up to roughly 4 epochs performs almost like fresh data, but returns fall off quickly beyond that. So for a small language, adding new unique text is worth more than upsampling what you have, and every genuinely new page counts.

Why it matters. You do not need to understand a thousand techniques. You need to ask of any pipeline step: is this mixing, filtering or deduplication, and how does it change the distribution the model will copy? The order in the diagram is also not sacred. Cheap, aggressive steps usually run first because they shrink the data before expensive steps touch it, and different teams arrange the same boxes differently. What never changes is the loop.

💻 Code: chapter-01/1_1_2_mixture_and_filters.py. Applies mixing weights, a chain of filters and deduplication to a toy corpus, then prints the distribution before and after each step so you can see every p_filtered change.

Expected code output:

1.1.3 How Big Is "Big"?

The idea. A pretraining run is measured in tokens, and a token is, on average, about 0.75 of an English word. Llama 3 was trained on roughly 15 trillion tokens. What does that number mean for a human?

The math. The conversion is a chain of unit changes:

                       tokens × 0.75 words per token
years of reading  =  ─────────────────────────────────────────────────
                     250 words per minute × 525,600 minutes per year

Read it as: turn tokens into words, then words into minutes of reading, then minutes into years.

Worked example.

1. 15 trillion tokens × 0.75 words per token ≈ 11.25 trillion words
2. 11.25 trillion words ÷ 250 words per minute ≈ 45 billion minutes
3. 45 billion minutes ÷ 525,600 minutes per year ≈ 85,600 years of non-stop reading

What survives cleaning is small. FineWeb-Edu, the heavily filtered educational subset of the 15-trillion-token FineWeb dataset, keeps about 1.3 trillion tokens. That is the keep rate Z from the last section:

       1.3 trillion
Z  =  ──────────────  ≈  0.087    (8.7%)
       15 trillion

For every 100 pages in the full dataset, the educational filter judges about 91 unfit and keeps about 9.

A hidden cost: your language may need more tokens for the same words. The "0.75 words per token" figure is for English. The general quantity is called fertility, the average number of tokens a tokenizer needs per word:

       tokens
φ  =  ────────        English:  φ ≈ 1 / 0.75 ≈ 1.33
       words

If a tokenizer was built mostly from English text, a language it sees rarely gets chopped into many small pieces, so its φ is higher. Suppose φ = 3 for some language (illustrative; you should measure your own). Three things follow directly:

Tokenization (Chapter 4) is therefore not a neutral technical step. It decides how expensive your language is to learn.

💻 Code: chapter-01/1_1_3_scale_and_fertility.py. Reproduces the 85,600-year calculation, then measures fertility φ for any text file you point it at, so you can compare English with a local language on the tokenizer of your choice.

Expected Output:

1.1.4 The Quantity, Quality, Diversity Trade-off, Measured

The idea. Quantity, quality and diversity pull against each other. Aggressive filtering raises quality but shrinks quantity and diversity. Keeping everything preserves quantity but drowns the model in noise. Every technique in this series is a negotiation between the three. Let's make that negotiation measurable.

The math. Give every document a quality score q(x) from a classifier, and keep the documents at or above a threshold τ. Then:

Quantity:   n(τ)  =  number of documents with q(x) ≥ τ

Quality:    q̄(τ)  =  average of q(x) over the documents with q(x) ≥ τ

Diversity:  H(τ)  =  − Σ_g  p_g · ln(p_g)

Read it as:

• Quantity is how many documents survive the cut.
• Quality is the average score among the survivors.
• Diversity is the entropy of the share p_g that each group g (language, topic, region) holds in the kept set. It is highest when groups are balanced and zero when one group is everything.

n(τ) can only go down as τ rises, and the average quality can only go up. Diversity can do either, and raising τ can silently destroy it.

Worked example. Take 10 documents: 6 English web pages, 2 forum threads and 2 Yoruba pages. A quality classifier trained on English reference text gives these scores:

Set τ = 0.5. Only documents scoring 0.5 or higher survive: the five best English pages and the 0.60 forum thread. The 0.30 English page, the 0.20 forum thread and both Yoruba pages are removed. Here is the before and after:

The two entropy values, step by step:

H_before  =  −( 0.6·ln 0.6  +  0.2·ln 0.2  +  0.2·ln 0.2 )      =  0.950

H_after   =  −( (5/6)·ln(5/6)  +  (1/6)·ln(1/6) )                =  0.451

Quality rose by 41% (0.767 ÷ 0.545 ≈ 1.41) and diversity fell by more than half. The Yoruba text was not low quality in any real sense. The classifier simply never learned what good Yoruba looks like.

Why it matters. FineWeb-Edu's 8.7% keep rate is quality bought at a steep, deliberate price in quantity. This example shows the second price: filters carry the biases of whatever they were trained on. Chapter 3 returns to this when we build filters that do not erase the languages we care about.

💻 Code: chapter-01/1_1_4_filter_tradeoff.py. Reproduces the table above, then sweeps τ from 0 to 1 and plots quantity, quality and diversity on one chart.

EXpected Output:

↑ Back to contents

Module 1.2: The Mathematics of the Training Objective

1.2.1 Entropy, Cross-Entropy and KL Divergence

The idea. Training a language model means minimizing one quantity: the average surprise the model feels at the real next word, measured across the dataset. Information theory gives this quantity a name, cross-entropy, and splits it into two pieces with very different meanings.

The math. Let p be the true distribution of the data and q the model's distribution. Define three quantities.

Entropy, the irreducible randomness in the data:

H(p)  =  − Σₓ p(x) · ln p(x)

Cross-entropy, the average surprise of a model q on data that actually comes from p:

H(p, q)  =  − Σₓ p(x) · ln q(x)

KL divergence, how far the model is from the data:

                          p(x)
KL(p ‖ q)  =  Σₓ p(x) · ln ────
                          q(x)

Read them as: entropy is the surprise you'd feel even with perfect knowledge. Cross-entropy is the surprise you actually feel with an imperfect model. KL divergence is the extra surprise caused by the model being wrong.

These three are linked by one short identity. Expand the logarithm inside the KL divergence:

KL(p ‖ q)  =  Σₓ p(x) ln p(x)  −  Σₓ p(x) ln q(x)

           =  −H(p)  +  H(p, q)

Rearranging gives the decomposition everything in this series rests on:

╔═══════════════════════════════════╗
║   H(p, q)  =  H(p)  +  KL(p ‖ q)  ║
╚═══════════════════════════════════╝

Read it as: total surprise = unavoidable surprise + surprise from being wrong.

Why the floor cannot move. KL divergence is never negative. The proof is short. It uses the fact that ln is concave (Jensen's inequality), which says Σ p · ln(r) ≤ ln(Σ p · r):

                          q(x)
−KL(p ‖ q)  =  Σₓ p(x) · ln ────
                          p(x)

             ≤  ln [ Σₓ p(x) · q(x)/p(x) ]          (Jensen)

             =  ln [ Σₓ q(x) ]   ≤   ln 1   =   0

So KL ≥ 0, with equality exactly when q = p. Therefore H(p, q) ≥ H(p): no model can score a cross-entropy below the entropy of the data.

Reading the two pieces:

Training pushes the model toward the data's distribution and nothing else. Read that with data quality in mind: the best possible outcome of training is a model that perfectly reproduces whatever is in your dataset.

💻 Code: chapter-01/1_2_1_entropy_kl_identity.py. Draws random distributions p and q, computes all three quantities, and checks H(p, q) = H(p) + KL to machine precision. Also checks that KL is never negative.

Expected Output:

1.2.2 Training Is Maximum Likelihood

The idea. In practice we never know the true p. We only have a finite dataset. So what does training actually minimize?

The math. From a dataset of n tokens, build the empirical distribution p̂(x) = n(x) / n, the fraction of the dataset equal to each outcome. The training loss is the average negative log-probability the model assigns to the data:

L(θ)  =  −(1/n) · Σᵢ ln q_θ(xᵢ)  =  H(p̂, q_θ)  =  H(p̂) + KL(p̂ ‖ q_θ)

Read it as: the training loss is the cross-entropy between your dataset and the model. Minimizing it, maximizing likelihood and minimizing KL to the dataset are three names for the same thing. The target is p̂: not "language", not "truth", but your dataset.

Worked example: what is the best possible model? Let a model be a single categorical distribution with probabilities q_k over K outcomes, and let n_k be how often outcome k appears. We want to maximize the data's log-likelihood subject to the probabilities adding up to 1. Use a Lagrange multiplier λ.

Step 1. Write the objective with the constraint attached:

J  =  Σⱼ nⱼ · ln qⱼ  −  λ · ( Σⱼ qⱼ − 1 )

Step 2. Set the derivative with respect to each q_k to zero:

∂J         nₖ
───  =    ──── − λ  =  0        ⟹        qₖ  =  nₖ / λ
∂qₖ        qₖ

Step 3. The probabilities must add up to 1, so Σ n_k / λ = n / λ = 1, which forces λ = n. Therefore:

q*ₖ  =  nₖ / n

The optimal model is simply the frequency table. This is exactly the bigram counter from 1.1.1, now proven to be the best possible solution of its kind. A neural network is a vastly more flexible, smoother way of approximating the same target.

Why it matters. "The model learns from data" sounds soft. This is the hard version: the loss function has the dataset's frequencies baked in as its minimum. Remove a document and the target changes. Duplicate a document and its weight in the target grows.

💻 Code: chapter-01/1_2_2_mle_is_frequency.py. Runs gradient descent on a categorical model and shows the learned probabilities converge to n_k / n.

Expected Output:

1.2.3 The Cost of Spam, Derived

The idea. Suppose a fraction ε of your dataset is spam. How much does that hurt a model that trains perfectly? We can answer this exactly, with no experiment.

The math. The data is a mixture of clean text and spam:

p_data(x)  =  (1 − ε) · p_clean(x)  +  ε · p_spam(x)

A perfect training run reaches q = p_data (from 1.2.2). Now measure that perfect model on clean held-out text. Assume spam and clean text look different, so on clean text only the first term matters: p_data(x) = (1-ε) · p_clean(x). The loss is:

L_clean  =  − Σₓ p_clean(x) · ln[ (1 − ε) · p_clean(x) ]

         =    H(p_clean)         +        ln( 1 / (1 − ε) )
              └──────────┘                └─────────────────┘
          best a clean-trained              the spam tax
              model can do

Because perplexity is e^L, the spam tax becomes a clean multiplier:

                                          1
perplexity inflation  =  e^(spam tax)  =  ─────
                                         1 − ε

Read it as: if 10% of your data is spam, a perfectly trained model is 1 ÷ 0.9 = 1.11 times more confused on clean text than a model trained on clean data alone.

Worked example.

The intuition is wasted probability. A model that has perfectly learned your data gives spam-shaped text a total probability of ε, so only 1 − ε is left for everything else. There is a second consequence worth more than the perplexity number: when you sample text from this model, a fraction ε of what it writes is spam. Pollution does not just make the model worse at measuring good text. It makes the model produce bad text at the rate it appeared in training.

A caveat for honesty. This is a best case. Real spam overlaps partly with ordinary language, and real models do not reach q = p_data exactly. The formula is a clean lower bound on the damage, not a full picture.

1.2.4 Proof in Code: Clean vs Spam-Polluted Training

The idea. Let's test the derivation with the smallest possible experiment. I built a character-level language model that predicts the next character using nothing but counts, and trained two copies on the same public-domain text: one clean, and one with spam lines inserted after roughly every quarter of the lines.

held-out loss, trained on clean text:          2.4838
held-out loss, trained on spam-polluted text:  2.6550

Both models were tested on the same clean held-out text. The math check: the loss gap is 2.6550 − 2.4838 = 0.1712 nats, so the perplexity ratio is

PPL_polluted
────────────  =  e^(2.6550 − 2.4838)  =  e^0.1712  ≈  1.187      (equivalently 14.2 / 12.0)
 PPL_clean

If the bound from 1.2.3 were tight, a multiplier of 1.187 would mean 1/(1−ε) = 1.187. Solve for ε:

             1
ε  =  1 − ───────  ≈  0.16
           1.187

So the polluted model behaves as though about 16% of its probability mass is tied up in spam-shaped patterns. Treat that as a rough reading, since a toy counter does not satisfy the clean separation assumption exactly, but the order of magnitude is what the theory predicts.

Sample text generated from the polluted model shows fragments of the inserted spam, words like CLICK, BUY, FREE and www, sitting inside what should have been ordinary prose. A transformer with billions of parameters is a far better copier than this toy model. It will learn the spam more precisely, not less.

💻 Code: chapter-01/1_2_4_spam_pollution_experiment.py. Trains both character models on a public-domain text file, prints held-out loss and perplexity, measures the real spam share ε, and compares the observed perplexity ratio against 1/(1−ε). Change the spam rate to reproduce the table in 1.2.3.

1.2.5 The ln(V) Sanity Check

The idea. At the very start of training, before the model has learned anything, it should know nothing. What loss does knowing nothing give?

The math. A model that knows nothing assigns the same probability to every token: q(x) = 1/V. Plug that into the cross-entropy. This works for any data distribution p:

H(p, q)  =  − Σₓ p(x) · ln(1/V)  =  ln V · Σₓ p(x)  =  ln V · 1  =  ln V

So the starting loss should sit almost exactly at ln V, and the starting perplexity at exactly V (the model is as unsure as if choosing among all V tokens).

Worked example.

How to use it as a debugging tool. Compare your first logged loss against ln V:

• Far above ln V: the model starts overconfident, often because initial weights are too large.
• Far below ln V: the model somehow already knows the answers. Suspect label leakage, where the targets are accidentally visible in the inputs, or a data bug that makes text trivially predictable (heavy duplication, for instance).
• Right at ln V but never falling: the model is not receiving a usable signal. Check that inputs and targets are properly aligned and that the data is not noise.

This single check has caught more pipeline bugs than almost any other. It costs one print statement.

💻 Code: chapter-01/1_2_5_ln_vocab_check.py. Computes ln V for common tokenizers and runs a randomly initialized model on a batch to confirm the starting loss lands where theory says.

What this means practically: the moment you accept a document into your training set, you are voting for the model to reproduce it. Every filtering decision in the rest of this series exists because of this one mathematical fact.

↑ Back to contents

Module 1.3: What the Model Is Truly Optimizing For

1.3.1 From Tokens to Loss to Perplexity

The idea. A document is a sequence of tokens, and its probability is the product of each token's probability given what came before. Products of many small numbers collapse toward zero, so in practice everything runs in log space, where multiplication becomes addition.

The math. From the chain rule in 1.1.1, take the logarithm and average over the T tokens. The flipped, averaged log-probability is the loss:

L  =  −(1/T) · Σₜ ln p_θ(xₜ | x<ₜ)        (add over t = 1 … T)

Perplexity is e raised to the loss:

PPL  =  e^L  =  ( ∏ₜ  1 / p_θ(xₜ | x<ₜ) ) ^ (1/T)

Read it as: perplexity is the geometric mean of the inverse probabilities. It has an intuitive reading: if the model were choosing uniformly among k options at every step, each probability would be 1/k and PPL = k. A perplexity of 15 means the model is, on average, about as unsure as if picking among 15 equally likely options.

Worked example. A four-token sentence where the model assigns probabilities 0.5, 0.25, 0.1 and 0.8 to the true tokens:

Then:

       4.6051
L  =  ────────  =  1.1513          PPL  =  e^1.1513  ≈  3.16
         4

Check it the other way, straight from the product: (1 ÷ 0.01)^(1/4) = 100^0.25 = 3.16. ✓

Why log space is not optional. A 1,000-token document where each token gets probability 0.1 has probability 10^-1000. The smallest number a 64-bit float can hold is about 10^-308. Multiplying directly gives zero, and zero's logarithm is minus infinity. Summing logs gives the stable answer:

Σ ln 0.1  (1,000 times)  =  1000 × (−2.3026)  =  −2302.6

A trap: perplexity is not comparable across tokenizers. Per-token loss depends on how text was chopped into tokens. A tokenizer that splits text into many tiny, easy-to-predict tokens gets a lower per-token loss without being a better model. The fair, tokenizer-independent measure is bits per byte (BPB):

        T × L
BPB  =  ────────
        B × ln 2

where T is the number of tokens, L the loss per token in nats, and B the number of bytes in the text. The ln 2 converts nats to bits.

Worked example. A document of 1,000 tokens and 4,200 bytes at L = 2.3 nats per token:

        1000 × 2.3         2300
BPB  =  ─────────────  =  ──────  ≈  0.790
        4200 × 0.6931      2911

When you compare your model against someone else's, compare BPB, not perplexity. This matters especially for African languages, where tokenizers differ most.

💻 Code: chapter-01/1_3_1_perplexity_and_bpb.py. Computes the worked example both ways, demonstrates the underflow failure of naive multiplication, and converts loss to bits per byte.

1.3.2 How the Data Pushes on the Model: the Gradient

The idea. Now we can look inside a single training step and see exactly how data shapes the model. A model outputs a score (a logit) z_i for each of the V tokens, and a softmax turns scores into probabilities.

The math. The softmax and the loss for one example, where y is the true next token:

        e^zᵢ
pᵢ  =  ────────          ℓ  =  −ln p_y  =  −z_y + ln Σⱼ e^zⱼ
       Σⱼ e^zⱼ

Differentiate the loss with respect to a logit z_i. The first term contributes −1 only when i = y, and the second term differentiates to the softmax probability itself:

╔══════════════════════════════════════╗
║   ∂ℓ / ∂zᵢ  =  pᵢ − 𝟙[ i = y ]       ║
╚══════════════════════════════════════╝

Read it as: the gradient is just what the model predicted minus what actually happened. (𝟙[i = y] is 1 for the true token and 0 for every other token.) Gradient descent moves each logit against this gradient: it raises the true token's score and lowers all the others in proportion to how much probability they stole.

Worked example. Three tokens with logits (2, 1, 0), and the true token is token 3. The softmax needs e² = 7.389, e¹ = 2.718, e⁰ = 1, which sum to 11.107:

The loss is −ln 0.090 = 2.41. The true token was given only 9% probability, so the push is strong. The gradient entries sum to exactly zero (0.665 + 0.245 − 0.910 = 0), because probability moved is probability conserved.

Averaging over the dataset. For a fixed context c, the average gradient over all the training examples that share that context is:

E_data[ ∇_z ℓ ]  =  q_θ(· | c)  −  p_data(· | c)

Read it as: the average push equals "what the model believes" minus "what the data says." This is zero only when the model's distribution matches the data's distribution for that context. The update never stops pulling until the model has copied the data. If spam follows context c with frequency ε, then spam tokens receive a pull proportional to ε on every step, for as long as training runs. This is the one-line mechanism behind "the model becomes whatever you feed it."

💻 Code: chapter-01/1_3_2_softmax_gradient_check.py. Verifies the formula p − onehot against numerical finite-difference gradients, reproduces the three-token example, and shows the averaged gradient going to zero as the model matches the data.

1.3.3 Scaling Laws, With Real Numbers

The idea. Scaling laws describe how loss falls as a model and its training data both grow. The widely cited Chinchilla work (Hoffmann et al., 2022) fits a simple form with three terms: a floor, a term that shrinks as the model grows, and a term that shrinks as the data grows.

The math. With N parameters and D training tokens:

L(N, D)  =    E     +    A / N^α    +    B / D^β
            ─────      ───────────     ───────────
            floor      model-size       data-size
                          term            term

The fitted constants reported in the paper's third estimation approach are approximately:

These constants belong to the paper's own dataset and tokenizer, so treat the absolute losses as illustrative rather than as predictions for your model. What transfers is the shape.

Worked example 1: what doubling the data buys. The data term scales as D^−β, so doubling D multiplies it by:

2^(−0.28)  ≈  0.824

Doubling your data shrinks the data-related part of the loss by about 18%. It never touches E.

Worked example 2: two real model sizes. Take Llama 3's 8B model at 15T tokens, and the 405B model at about 15.6T tokens. Each term, computed from the formula:

Look at the 405B row. The model-size term has shrunk to 0.046, while the data term (0.083) is nearly double it. For a model that large, data is the term still holding the loss up. This is the mathematical reason the field's appetite for clean, usable text keeps climbing.

Where data quality enters. The formula counts tokens, but not all tokens are equal. Better filtering behaves like a larger effective D, or equivalently a smaller B. This is the quantitative form of "a smaller, cleaner dataset can beat a larger, dirtier one."

💻 Code: chapter-01/1_3_3_chinchilla_calculator.py. Implements L(N, D) with the fitted constants, reproduces both rows of the table, and plots loss curves as you vary N and D.

1.3.4 Compute-Optimal vs Over-Trained

The idea. Given a fixed compute budget, how should you split it between a bigger model and more data?

The math. Training compute is approximately 6 floating-point operations per parameter per token:

C  ≈  6 · N · D

Substitute D = C / (6N) into L(N, D) and set the derivative with respect to N to zero. The result is that the best N and D both grow with compute, as power laws:

N_opt  ∝  C^( β / (α+β) )  =  C^( 0.28 / 0.62 )  ≈  C^0.45

D_opt  ∝  C^( α / (α+β) )  =  C^( 0.34 / 0.62 )  ≈  C^0.55

Read it as: when your budget grows, spend it on a bigger model and more data, at close to the same rate. The practical rule of thumb that came out of this is roughly 20 training tokens per parameter.

Worked example: how far modern models sit from that rule.

Why anyone over-trains. The "optimal" above minimizes loss for a fixed training budget. But a model is trained once and used many times. Inference costs roughly 2 operations per parameter per token served, so with T tokens served over a model's lifetime:

inference cost       2 · N · T         T
──────────────  =  ───────────  =  ─────
training cost       6 · N · D        3 D

The model size N cancels out. If a model serves more than 3D tokens in its lifetime, inference dominates the bill. For D = 15T, that threshold is 3 × 15T = 45 trillion tokens served, which a widely deployed model can pass. So it pays to train a smaller model on far more data: slightly worse for the training bill, much cheaper for everyone who runs it afterward. That single design choice is why a smaller model's hunger for data keeps rising even as model sizes stay similar. It is also good news for builders with limited compute: a small model trained on excellent data is exactly the shape of model we can afford to run.

💻 Code: chapter-01/1_3_4_compute_optimal_and_inference.py. Finds the loss-minimizing N and D for a given compute budget, and finds the break-even point where inference cost passes training cost.

1.3.5 Where the Numbers From the Intro Come Back

This is where the opening numbers of the post connect back to the mathematics. Meta's published figures say training the 405-billion-parameter model consumed 30.84 million GPU-hours. Check that against C ≈ 6ND.

Step 1: total compute needed.

C  ≈  6 × (405 × 10⁹) × (15.6 × 10¹²)  ≈  3.8 × 10²⁵ FLOPs

Step 2: total GPU-seconds available.

30.84 × 10⁶ hours × 3600 s/hour  ≈  1.11 × 10¹¹ GPU-seconds

Step 3: compute delivered per GPU per second.

 3.8 × 10²⁵
────────────  ≈  3.4 × 10¹⁴ FLOP/s per GPU
1.11 × 10¹¹

An H100's peak throughput in the precision used for training is on the order of 10^15 FLOP/s, so each GPU ran at roughly a third of peak, which is in the normal range for large training runs. The introduction's numbers are not exaggerations; they follow from the same arithmetic as the equation above.

All of that compute exists to shrink one number, the loss, by pushing the model closer to the data it was shown. In the language of 1.2.1, it drives KL(p̂ ‖ q_θ) toward zero and leaves H(p) exactly where it is. None of it can make the model smarter than its data. A perfectly optimized model trained on a flawed dataset is still a flawed model, just a very confident one.

What you can do on a laptop. You cannot afford 30.84 million GPU-hours. You can afford to measure the data term B / D^β for your own data. Train a tiny model on 1%, 2%, 4%, 8% of your corpus, plot held-out loss against tokens, and fit a line on log-log axes. The slope is your β. Compare two versions of your pipeline (before and after a filter) and you have a number for whether the filter helped. That is the loop from 1.1.2, with math attached.

💻 Code: chapter-01/1_3_5_mini_scaling_experiment.py. Trains small models on growing slices of a text file, fits L ≈ E + B / D^β on the results, and compares two dataset versions side by side.

↑ Back to contents

Chapter 1 Cheat Sheet

Try it yourself. Four exercises, solutions in the repo under chapter-01/exercises/:

1. In 1.1.1, how many spam copies make P(free | the) exceed 0.9?
2. A dataset is 8% spam. By what factor does a perfect model's perplexity on clean text rise? What fraction of its generated text will be spam?
3. A tokenizer has V = 65,536. What starting loss should you expect?
4. In 1.3.3, how many tokens would the 8B model need to push the data term B / D^β below 0.05?

References

• Meta AI, Llama 3.1 Model Card: official source for the 30.84 million GPU-hour, 700W and CO2 figures, and used again in 1.3.5.
• TIME, "Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic", January 18, 2023: the original investigation, based on payslips and internal documents.
• CBS News, "Kenyan workers with AI jobs thought they had tickets to the future until the grim reality set in", 60 Minutes, 2025: confirms the $12.50-to-$2 pay gap and separately implicates Scale AI's Remotasks platform.
• Alexandr Wang, Wikipedia: background on Scale AI and Meta's 2025 investment.
• Fox Business, "Trump rebrands AI, rejects 'globalist scheme' to control tech": coverage of the September 2025 United Nations speech and the superintelligence rebrand.
• Llama Team, AI @ Meta, "The Llama 3 Herd of Models", 2024: data mix proportions, token counts and training setup.
• Hoffmann et al., "Training Compute-Optimal Large Language Models" (the Chinchilla paper), 2022: the loss form and fitted constants used in 1.3.3 and 1.3.4.
• Kaplan et al., "Scaling Laws for Neural Language Models", 2020: the earlier scaling-law study Chinchilla revised.
• Muennighoff et al., "Scaling Data-Constrained Language Models", 2023: the repeated-epochs result in 1.1.2.
• Penedo et al., "The FineWeb Datasets", 2024: FineWeb and FineWeb-Edu sizes used in 1.1.3.

↑ Back to contents

Next up, Chapter 2: From the Open Web to a Clean Corpus.