This post was originally published on my blog: nitayneeman.com
How do Large Language Models (LLMs) like GPT, Claude and Gemini actually work? How does a text prompt turn into a coherent response?
Software development in 2026 relies heavily on AI tools that write code and even generate entire applications. These tools are powered by Large Language Models (LLMs), which are neural networks trained on massive amounts of text data that can understand and generate human language. Model families like GPT, Claude and Gemini are all LLMs. The "Large" word in the name refers to both the enormous training data and the billions of learnable parameters that make up the model.
There's no magic involved with LLMs - it all comes down to linear algebra, probability and a very clever architecture behind.
Every time we type a prompt and get a response, whether in a chat interface or a terminal, the model runs a series of steps called the inference pipeline:
In this article, we'll explain this pipeline and how it all works together.
Tokenization
An LLM doesn't read text the way we do - it breaks the input into tokens. Tokens are basically subword units that may correspond to whole words, parts of words, punctuation or spaces.
That's important because it allows the LLM ("the model") to identify deep language patterns rather than just memorizing whole words, enabling it to process new or complex vocabulary by reconstructing it from pieces it already knows.
For example, the sentence "How does AI work?" might become these tokens:
["How", " does", " AI", " work", "?"]
Most modern LLMs use an algorithm called Byte Pair Encoding (BPE) to break words into tokens.
BPE starts with individual characters and then iteratively merges the most frequent adjacent pairs into a single token. After thousands of merges, we end up with a vocabulary of ~50,000-100,000 subword tokens that balance between full words (common ones like "the") and fragments (like "un" + "expect" + "edly").
It also explains why LLMs sometimes behave strangely with spelling or certain words - they only see token chunks and not individual characters (for example, they may process "strawberry" as ["straw", "berry"] rather than as s-t-r-a-w-b-e-r-r-y).
Let's see a simplified version of how BPE works:
// Simplified BPE
function trainBPE(corpus, numMerges) {
// Starts with individual characters as our initial tokens
let tokens = corpus.split('').map(ch => ch);
for (let i = 0; i < numMerges; i++) {
// Counts every adjacent pair
const pairs = {};
for (let j = 0; j < tokens.length - 1; j++) {
const pair = tokens[j] + tokens[j + 1];
pairs[pair] = (pairs[pair] || 0) + 1;
}
// Finds the most frequent pair
const best = Object.entries(pairs).sort((a, b) => b[1] - a[1])[0];
if (!best) break;
const [merged, count] = best;
// Merges all occurrences of that pair into a single token
const next = [];
let j = 0;
while (j < tokens.length) {
if (j < tokens.length - 1 && tokens[j] + tokens[j + 1] === merged) {
next.push(merged);
j += 2;
} else {
next.push(tokens[j]);
j++;
}
}
tokens = next;
console.log(`Merge #${i + 1}: "${merged}" (appeared ${count} times)`);
}
return tokens;
}
trainBPE('the cat sat on the mat', 5);
// Merge #1: "at" (appeared 3 times)
// Merge #2: "th" (appeared 2 times)
// Merge #3: "the" (appeared 2 times)
// Merge #4: "the " (appeared 2 times)
// Merge #5: "at " (appeared 2 times)
In practice, real tokenizers like tiktoken (the official OpenAI's BPE tokenizer) operate on bytes and handle tens of thousands of merges - but the core algorithm is pretty much as above.
Although BPE is by far the most common in the GPT/Claude families, other approaches exist too: such as WordPiece and SentencePiece that are being used by Gemini models.
Once the text is broken into these tokens, the model has a list of integers. But these numbers alone don't mean anything and that's why each one gets mapped to an embedding.
Embeddings
An embedding is a high-dimensional vector mapped to each token, in order to capture what the token represents. These embeddings live in a high-dimensional vector space where relationships between words are encoded as distances and directions.
For example, words that appear in similar contexts like 'king' and 'queen' end up with vectors pointing in a similar direction. Their "dot product" (a measure of similarity) is high. Meanwhile, 'king' and 'banana' point in very different directions to result in a low dot product.
The model learns all these word relationships purely from patterns in the text.
// In a real model, embeddings are thousands of dimensions and learned during training
// Here we'll use small 4D vectors for clarity
const embeddingTable = {
'king': [0.9, 0.8, 0.2, 0.1],
'queen': [0.8, 0.9, 0.2, 0.2],
'banana': [0.1, 0.1, 0.9, 0.7],
'apple': [0.2, 0.1, 0.8, 0.8],
};
function embed(tokens) {
return tokens.map(token => embeddingTable[token]);
}
// Each token becomes a vector of numbers
embed(['king', 'queen', 'banana']);
// [
// [0.9, 0.8, 0.2, 0.1], ← "king"
// [0.8, 0.9, 0.2, 0.2], ← "queen"
// [0.1, 0.1, 0.9, 0.7], ← "banana"
// ]
// We can measure similarity using dot product
function dotProduct(a, b) {
return a.reduce((sum, val, i) => sum + val * b[i], 0);
}
// "king" and "queen" vs "king" and "banana"
dotProduct(embeddingTable['king'], embeddingTable['queen']); // 1.50
dotProduct(embeddingTable['king'], embeddingTable['banana']); // 0.42
// → "king" is much closer to "queen" than to "banana" in this space
The important thing is that these embeddings aren't created manually; they're learned during exposure to billions of sentences following a simple rule - words used in similar contexts will relate and be closer together.
Positional Encoding
There's a problem though - the model processes all tokens in parallel. That means without some extra information, the model has no idea whether a word appears at the beginning or end of a sentence (so "dog bites man" and "man bites dog" would look identical).
The solution is named positional encoding and extends each token's embedding with additional vectors encoding its position in the sequence.
// The original Transformer used sinusoidal functions:
function positionalEncoding(position, dim) {
const encoding = new Array(dim);
for (let i = 0; i < dim; i++) {
const angle = position / Math.pow(10000, (2 * Math.floor(i / 2)) / dim);
encoding[i] = i % 2 === 0 ? Math.sin(angle) : Math.cos(angle);
}
return encoding;
}
// Each position gets a unique pattern
positionalEncoding(0, 4); // [0.000, 1.000, 0.000, 1.000]
positionalEncoding(1, 4); // [0.841, 0.540, 0.010, 0.999]
positionalEncoding(2, 4); // [0.909, -0.416, 0.020, 0.999]
// Adds position info to the embedding
function addPosition(embeddings) {
return embeddings.map((emb, pos) => {
const pe = positionalEncoding(pos, emb.length);
return emb.map((val, i) => val + pe[i]);
});
}
// Reuses the embeddings from the previous example
const tokenEmbeddings = embed(['king', 'queen', 'banana']);
const positionedEmbeddings = addPosition(tokenEmbeddings);
console.log(positionedEmbeddings);
// [
// [0.900, 1.800, 0.200, 1.100], // "king" + position 0
// [1.641, 1.440, 0.210, 1.199], // "queen" + position 1
// [1.009, -0.316, 0.920, 1.699], // "banana" + position 2
// ]
Important to note that modern LLMs use Rotary Position Embeddings (RoPE), which encode relative distances between tokens (rather than absolute positions). That relative structure is one reason RoPE-based models can often generalize better to longer sequences than models with absolute positional embeddings.
The Transformer Architecture
"The Transformer" is a mathematical architecture behind every major LLM. It was introduced in the 2017 paper "Attention Is All You Need" and is designed to process entire sequences of data at once rather than word by word.
While embeddings assign each token a fixed vector regardless of context, the Transformer refines these vectors, allowing the same word to carry different meanings in different contexts.
Let's understand how it works.
Self-Attention
The Transformer's fundamental operation is called self-attention. It's the mechanism that transforms static embeddings into representations with context. For each token in the input, it looks at the entire sequence to determine which other tokens matter most for understanding this one.
Consider the following sentence:
"The animal didn't cross the street because it was too tired."
When the model processes the token "it", the self-attention mechanism calculates that "it" has a strong relationship with "animal" but a weak relationship with "street." So the model updates the vector for "it" to include information about the "animal."
The model uses weights (the "learnable parameters" that are learned during training) to project each token embedding into three vectors:
- Query (Q) - what am I looking for?
- Key (K) - what do I contain?
- Value (V) - what information do I carry?
These three vectors are necessary to calculate the attention scores (the relationships between tokens) based on the following formula:
Let's implement it:
function softmax(arr) {
const max = Math.max(...arr);
const exps = arr.map(x => Math.exp(x - max)); // subtracts max for numerical stability
const sum = exps.reduce((a, b) => a + b, 0);
return exps.map(x => x / sum);
}
function scaledDotProductAttention(Q, K, V) {
const dim = Q[0].length;
const scale = Math.sqrt(dim);
// For each query token, compute attention over all key tokens
return Q.map(q => {
// Score = dot product of query with each key, scaled
const scores = K.map(k =>
q.reduce((sum, val, i) => sum + val * k[i], 0) / scale
);
// In generation: apply causal mask before softmax (upper triangle = -Infinity)
// This ensures each token can only attend to previous tokens
// Normalizes scores into a probability distribution
const weights = softmax(scores);
// Output = weighted sum of values
return V[0].map((_, i) =>
weights.reduce((sum, w, j) => sum + w * V[j][i], 0)
);
});
}
// Example: 3 tokens, each with dimension 4
const Q = [[1, 0, 1, 0], [0, 1, 0, 1], [1, 1, 0, 0]];
const K = [[1, 0, 1, 0], [0, 1, 0, 1], [0, 0, 1, 1]];
const V = [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]];
// Note: Q, K, V are derived from learned weight matrices applied to the input
// In practice: Q = input * Wq, K = input * Wk, V = input * Wv
const output = scaledDotProductAttention(Q, K, V);
// Each token now contains a weighted mix of all values,
// weighted by how relevant each other token is to it
Note that each token can only attend to tokens that came before it in the sequence, never to future tokens. This prevents the model from "peeking ahead" at tokens it hasn't seen yet. In the code above, this would mean masking out the upper triangle of the attention scores matrix before applying softmax.
The whole process runs across multiple attention heads in parallel (called multi-head attention), where each head learns to track a different type of relationship: syntactic structure, coreference and semantic similarity.
// Example: 2 heads, dimension 4 → each head operates on 2 dimensions
function multiHeadAttention(input, numHeads = 2) {
const dim = input[0].length;
const headDim = dim / numHeads;
// Each head gets its own slice of the embedding
const heads = Array.from({ length: numHeads }, (_, h) => {
const start = h * headDim;
const end = start + headDim;
// Simplified: slices input directly. In practice, each head applies
// its own learned weight matrices (Wq, Wk, Wv) to project into Q, K, V
const Q = input.map(token => token.slice(start, end));
const K = input.map(token => token.slice(start, end));
const V = input.map(token => token.slice(start, end));
return scaledDotProductAttention(Q, K, V);
});
// Concatenates all head outputs back together
return input.map((_, i) =>
heads.flatMap(head => head[i])
);
}
Self-attention is the core of the Transformer, but it's not enough on its own. In practice, it's wrapped inside a larger structure called a "Transformer block", which adds the supporting layers needed to actually make it work at scale.
The Transformer Block
A "Transformer block" is a self-contained unit that gets repeated - each one follows the same internal pattern:
First, the input is normalized using Layer Normalization (modern LLMs typically use RMSNorm). This rescales values to a consistent range, preventing small numerical fluctuations from compounding across dozens of layers.
The output is added back to the original input via a residual connection (a shortcut that lets the original signal pass through unchanged. Also known as "skip connection") - this is critical for training deep networks, since without it, the learning signal ("gradients") would vanish long before reaching the early layers.
Then the same pattern repeats: normalize, pass through a Feed-Forward Network (FFN) - a two-layer neural network applied independently to each token, and add another residual connection. The FFN is basically where much of the model's factual knowledge is stored.
Let's put the whole block together:
function rmsNorm(x, epsilon = 1e-6) {
const rms = Math.sqrt(x.reduce((sum, val) => sum + val * val, 0) / x.length + epsilon);
return x.map(val => val / rms);
}
function feedForward(x, weights) {
// Two-layer network with ReLU activation (modern LLMs typically use SwiGLU or GELU)
// Layer 1: expands to 4x the dimension
const hidden = weights.W1.map(row =>
Math.max(0, row.reduce((sum, w, i) => sum + w * x[i], 0)) // ReLU
);
// Layer 2: projects back to original dimension
return weights.W2.map(row =>
row.reduce((sum, w, i) => sum + w * hidden[i], 0)
);
}
function addVectors(a, b) {
return a.map((val, i) => val + b[i]);
}
// All weight matrices (attn, ffn) are learned during training
function transformerBlock(input, weights) {
// Step 1: Normalize → Attention → Residual
const normed1 = input.map(token => rmsNorm(token));
const attended = multiHeadAttention(normed1, weights.numHeads);
const afterAttn = input.map((token, i) => addVectors(token, attended[i]));
// Step 2: Normalize → FFN → Residual
const normed2 = afterAttn.map(token => rmsNorm(token));
const ffnOut = normed2.map(token => feedForward(token, weights.ffn));
const output = afterAttn.map((token, i) => addVectors(token, ffnOut[i]));
return output;
}
Stacking Blocks into a Model
A single Transformer block captures some context through self-attention, but real language understanding requires more. Modern LLMs stack dozens of these blocks, each one refining the representation further. Over those layers, a hierarchy of abstraction emerges:
- Early layers capture surface patterns like punctuation and word forms
- Middle layers encode syntax and relationships between words
- Deep layers handle high-level semantics and task-specific behavior
After passing through those layers, each token's embedding is transformed from a simple word vector into a contextual representation - encoding the meaning of that token in the context of the entire input.
One thing to keep in mind - the model can only process a fixed number of tokens at once, which is known as the context window. That limit (ranging from a few thousand to over a million tokens in recent models) is set during training and determines how much text the model can "see" in a single pass.
Everything we described (attention, normalization, residual connections) happens within this window. If the input exceeds this limit, the model can't process it (the common context length exceeded error we may encounter when working with LLMs).
Now that the model has a rich and contextualized vector for each token, it can move on to the final step - predicting the next token.
Next-Token Prediction
The pipeline from tokenization to generation runs every time the model generates a response. This process of running the model on an input to produce an output is called inference, and at its core is next-token prediction:
Given a sequence of tokens that came before, the model predicts what comes next.
The prediction is done in this way:
- The model takes the last token's vector and projects it across the entire vocabulary to produce raw scores called logits.
- Then applies softmax (the same function we used in attention) to convert those logits into a probability distribution - a score for each token representing how likely it is to come next.
For a prompt like "The capital of France is," the distribution might look something like:
// "Paris" → 92.4%
// "located" → 2.1%
// "the" → 1.3%
// "a" → 0.8%
// ...
But the model doesn't simply pick the highest-probability token every time - that would make it deterministic and repetitive.
Here's where sampling comes in.
Sampling Strategies
Sampling means the model selects a token based on the probability distribution. As it sounds, higher-probability tokens are more likely to be picked (but obviously not guaranteed).
The way the model samples is what controls the balance between accuracy and creativity, and three main parameters control this balance:
Temperature - controls confidence. A low value (like 0.1) sharpens the distribution, making the model stick to its top choices. A high value (like 1.5) flattens it out, letting less likely tokens through (this is essentially where "creativity" comes from). At temperature = 0, the model always picks the top token (greedy decoding).
Top-k - simply cuts off everything below the top k tokens. If k=10, only the 10 most likely tokens are considered (regardless of their actual probabilities).
Top-p - keeps the smallest set of tokens whose combined probability reaches a threshold (let's say 0.9). When the model is confident, this might be just 2-3 tokens; when uncertain, it could be dozens.
In practice, these strategies are combined - a typical setup might apply temperature first to shape the distribution and then use top-p to cut the long tail.
The Generation Loop
Once the model picks a token, it appends that token to the input and runs the entire inference again (embeddings, attention, next-token prediction) to produce the next token.
This flow is called autoregressive generation - each token is conditioned on everything that came before it:
function generate(prompt, model, maxTokens = 50) {
let tokens = tokenize(prompt);
const result = [...tokens];
for (let step = 0; step < maxTokens; step++) {
// 1. Embeds tokens + adds positions
let hidden = embed(tokens);
hidden = addPosition(hidden);
// 2. Runs through all transformer layers
hidden = runTransformer(hidden, model.layers);
// 3. Gets logits for the LAST token only (that's our prediction)
const lastHidden = hidden[hidden.length - 1];
const logits = model.outputProjection(lastHidden);
// 4. Samples the next token
let probs = applyTemperature(logits, 0.7);
probs = topPSampling(probs, 0.9);
const nextToken = sampleFromDistribution(probs);
// 5. Appends and continue
result.push(model.vocab[nextToken]);
tokens = [...tokens, nextToken];
// Stops if we hit an end-of-sequence token
if (model.vocab[nextToken] === '<eos>') break;
}
return result.join(' ');
}
Every response we receive from an LLM is built exactly this way (one token at a time, over and over, until the model emits a stop signal).
In practice, this process is optimized using a KV cache - instead of recomputing attention over all previous tokens from scratch at each step, the model caches the key and value vectors from prior steps and only computes the new token's attention. That's what makes generation fast enough to feel interactive, despite the massive computation involved.
We've now covered the full pipeline - from tokenization to generation. The missing piece is how the model learns to do all of this.
Training
Training is the process that turns a neural network with random weights into a capable language model.
All those weights we discussed (projection matrices in attention, feed-forward networks) start as random numbers and are learned through training. Once training completes, the weights are frozen and don't change (even when we use the model). The way to update them is to train a new model from scratch (GPT-4 → GPT-5 → GPT-6, Claude 3 → Claude 4 → Claude 5 and so on).
This training process is split into two major phases.
Pre-training
During this phase, the model runs the same inference pipeline we described earlier (tokenization, embeddings, attention, next-token prediction) - but on massive amounts of text: books, code, articles and documentation.
After each prediction, the model compares its output to the actual next token and adjusts its weights to reduce the error. The adjustment happens through backpropagation using an optimization process called "gradient descent". Through billions of these cycles, the model implicitly learns grammar, reasoning patterns, code syntax and much more.
Models like GPT-4 and Claude have hundreds of billions of these weights, trained on trillions of tokens, using thousands of GPUs over months. The compute cost for a single training run can reach tens of millions of dollars.
Post-training - Alignment
Pre-training gives us a model that can predict text, but that's not enough. A model that just completes text isn't necessarily helpful or safe. So after pre-training completes, the second phase begins: post-training.
The purpose of post-training is to align the model with human preferences so it actually follows instructions and gives useful answers.
At first, RLHF (Reinforcement Learning from Human Feedback) was the main approach and worked roughly like this: humans ranked model outputs from best to worst, a reward model learned those preferences and then the LLM was fine-tuned to maximize the reward.
Since then, the field has moved toward simpler approaches:
DPO (Direct Preference Optimization) - skips the reward model entirely and directly optimizes the LLM on human preference pairs. It's simpler, more stable and increasingly popular.
Constitutional AI (CAI) - developed by Anthropic (the company behind Claude). Instead of relying only on human labelers, the model critiques and revises its own outputs based on a set of principles. It scales better and makes the alignment process more transparent.
Post-training is what turns a text-completion engine into something that follows instructions, avoids harmful outputs and writes structured code when we ask it to.
The next challenge is making all of this run efficiently at scale with hundreds of billions of weights.
Scaling Up
Mixture of Experts (MoE)
Training and running a dense model with hundreds of billions of weights is extremely expensive - every token passes through every weight. "Mixture of Experts" offers a clever alternative.
Instead of one massive feed-forward network per layer, we use multiple smaller "expert" networks and a lightweight router that decides which experts to activate for each token.
Typically, only two out of the experts are active per token:
function mixtureOfExperts(x, experts, router) {
// Router decides which 2 experts handle this token
const routerScores = softmax(
router.weights.map(row => row.reduce((sum, w, i) => sum + w * x[i], 0))
);
// Picks top 2 experts
const topExperts = routerScores
.map((score, i) => ({ score, i }))
.sort((a, b) => b.score - a.score)
.slice(0, 2);
// Weighted combination of expert outputs
const totalWeight = topExperts.reduce((sum, e) => sum + e.score, 0);
return x.map((_, dim) =>
topExperts.reduce((sum, { score, i }) => {
const expertOut = feedForward(x, experts[i]);
return sum + (score / totalWeight) * expertOut[dim];
}, 0)
);
}
So the model can have a very large total weight count (for capacity) while only using a fraction of them per prediction step (for speed). GPT-4 is widely believed to use a MoE architecture and open-source models like Mixtral have demonstrated the approach openly.
MoE tackles the cost of scale. But scaling isn't just about efficiency - it's also about what the model can process.
Multimodal Models
Modern LLMs aren't limited to text anymore. Models like Gemini or Claude can process images, and some can handle audio and video as well.
The general approach is to use a separate encoder (like a Vision Transformer) to convert images into the same embedding space the LLM operates in. These visual tokens get interleaved with text tokens and the Transformer processes them together.
From the model's perspective, an image is just another sequence of tokens. Audio and video follow a similar pattern - each modality has its own encoder that produces tokens the Transformer can process.
Scaling up is one direction, but reducing is also important.
Quantization
"Quantization" is a technique for reducing the precision of a model's weights - converting them from high-precision numbers (16-bit or 32-bit) to lower precision (8-bit, 4-bit or even lower). That cuts memory requirements significantly with a small hit to quality.
A 70B-parameter model in full precision needs ~140GB of memory. Quantized to 4-bit, it fits in ~35GB - suddenly runnable on a high-end consumer GPU.
Tools like llama.cpp, Ollama and vLLM make this practical. The quality trade-off depends on the quantization method, but modern approaches (like GPTQ and AWQ) are very good at preserving model performance.
What is Built on Top
Everything we've covered so far describes the inference pipeline - from tokenization to next-token prediction. The tools we use daily (Claude, Cursor, etc.) layer additional techniques on top of that core.
Here are some of the most common ones:
Function calling (also called "Tool Use") - the model is trained to recognize when it should call an external tool (run a shell command, search the web, read a file) instead of generating text. It outputs a structured function call, the system executes it and the result gets fed back into the conversation.
Chain of Thought - by letting the model "think" step by step before answering, accuracy on complex tasks improves significantly. Some models (like Claude) have explicit extended thinking, where they work through a problem internally before producing a final answer.
RAG (Retrieval-Augmented Generation) - instead of relying only on what was learned during training, relevant documents are retrieved at inference time (when the user sends a prompt) and injected into the model's context. This is basically how AI coding tools understand our specific codebase.
Agents - an orchestration layer that runs the model in a loop: think → act → observe → repeat. Instead of one-shot generation, the model can plan multi-step tasks, execute them, handle errors and iterate.
None of these techniques change the model's architecture - they're all built around the same inference pipeline.
Wrapping Up
We covered in this article the inference pipeline behind Large Language Models - from how text becomes tokens to how the next token is predicted.
Here's a summary:
- LLMs are neural networks trained on massive text data, with billions of weights ("learnable parameters")
- Text is broken into subword tokens using algorithms like BPE, typically producing a vocabulary of ~50,000-100,000 tokens
- Each token is mapped to a high-dimensional embedding vector that captures its meaning
- Positional encoding is added to embeddings so the model knows word order
- The Transformer architecture refines these vectors through self-attention, letting each token attend to tokens that came before it
- A Transformer block combines normalization, attention, a feed-forward network and residual (skip) connections
- Modern LLMs stack 80-120+ of these blocks, with early layers capturing syntax and later layers handling semantics
- The process of running the model is called inference - at its core, the model predicts the next token, with sampling strategies like temperature, top-k and top-p controlling the output
- Pre-training teaches the model to predict text; post-training (RLHF, DPO, CAI) aligns it with human preferences
- Techniques like MoE, multimodal encoding, quantization, tool use, RAG and agents are built on top of this core loop
The better we understand this pipeline, the better we can work with it - whether that means writing precise prompts or building our next layer on top.
Here's an interactive project with all examples:
You can also find the source code on GitHub.
This post was originally published on nitayneeman.com. Follow me there for more deep dives into web development and AI.
