Embeddings: Meaning as Numbers

When you ask a chatbot for a pasta recipe, it doesn’t understand “pasta” the way you do. It sees a list of numbers. Those numbers are not random. They are arranged so that “pasta” sits close to “spaghetti” and far from “car.” This is an embedding. It turns meaning into coordinates that a machine can measure, compare, and search. Every time a chatbot finds a relevant answer, recommends a product, or remembers a fact from earlier in the conversation, embeddings are doing the heavy lifting behind the scenes.

Here is the surprising part. The machine never learns what “pasta” means. It only learns which words appear in the same kinds of sentences. That statistical shadow turns out to be so rich that it captures everything from synonyms to analogies. The rest of this article unpacks how that happens, step by step, using the same chatbot interactions you already know from this series.

What exactly is an embedding?

An embedding is a list of numbers that represents something discrete, a word, a token, a user ID, a product. Each number is a coordinate in a high-dimensional space. If you pick the right coordinates, similar things end up near each other. That is the entire idea.

Think of a map. A city’s latitude and longitude don’t tell you its name or history. But if you know that Paris is at (48.9, 2.3) and Lyon is at (45.8, 4.8), you can measure the distance and see they are both in France. Embeddings work the same way, except they use hundreds of dimensions instead of two. Every dimension captures some latent feature of the input, learned from data. The model never labels those features. It just arranges points so that words that behave similarly in text end up with similar coordinates.

In a chatbot, every token from Part 2 gets its own embedding vector. The model then uses those vectors as the starting point for everything else, including the attention mechanism from Part 3. The quality of the embeddings directly determines how well the model can tell that “I need a quick dinner idea” and “fast evening meal suggestions” mean roughly the same thing.

How does a chatbot turn words into numbers?

You already know that the chatbot breaks your message into tokens. For the query “give me a pasta recipe,” the tokenizer might produce token IDs like [123, 45, 678, 901]. Those integers are just labels. They carry no meaning on their own.

The model has a large table called an embedding matrix. It has one row for every token in the vocabulary. Each row is a vector of, say, 768 numbers. When the model sees token ID 678, it looks up row 678 and pulls out that vector. This lookup is the embedding layer. It turns a sequence of token IDs into a sequence of dense vectors that the rest of the network can process.

graph TD
A["Input text 'pasta recipe'"] --> B["Tokenizer produces IDs"]
B --> C["Embedding matrix lookup"]
C --> D["Vector for token 'pasta'"]
C --> E["Vector for token 'recipe'"]

This lookup is fast and differentiable. During training, the model adjusts the numbers in those rows so that the vectors become useful for the task at hand. If the model is trained to predict the next word, then words that lead to similar next-word predictions will gradually drift toward each other in the vector space. The embedding layer itself has no built-in notion of meaning. It is just a giant spreadsheet of numbers that gets updated by backpropagation.

The same mechanism applies to any discrete input. When a recommendation system sees your user ID, it fetches a vector that represents your preferences. When a search engine indexes a document, it stores an embedding vector for the whole document. The table lookup is the universal first step for turning symbols into numbers a neural network can digest.

Why do similar words end up with similar vectors?

The key insight is the distributional hypothesis. Words that appear in similar contexts tend to have similar meanings. If you see the word “pasta” surrounded by “sauce,” “boil,” “dinner,” and “recipe,” and you see “spaghetti” in the exact same kinds of sentences, then a model that learns from those contexts will place “pasta” and “spaghetti” close together. It doesn’t know what either word means. It only knows they are interchangeable in many sentences.

Early embedding methods made this explicit. They would scan a huge corpus and count how often every pair of words appeared near each other. That produced a giant co-occurrence matrix. Then they would squash that matrix down to a small number of dimensions using a technique like singular value decomposition. The result was a compact vector for each word that preserved the most important co-occurrence patterns. This is like taking a huge spreadsheet of word relationships and compressing it into a few columns that capture the gist.

Modern models do something similar, but they learn the vectors on the fly while training a neural network. They don’t count everything first. Instead, they look at a small window of words, try to predict a target word from its neighbors (or vice versa), and adjust the vectors to get better at that prediction. Over millions of examples, the vectors settle into a geometry where words that predict the same contexts cluster together.

This is why embeddings can solve analogies. The classic example is “king minus man plus woman equals queen.” The vectors don’t store royal titles. They store the directions that separate gender and royalty, learned from thousands of sentences where these words appear in parallel roles.

How does the model learn these vectors?

The most famous example is Word2Vec’s skip-gram model with negative sampling. The idea is simple. Take a sentence like “I need a quick pasta recipe.” Slide a window over it. For each center word, like “pasta,” pick a nearby context word, like “recipe.” The model’s job is to decide whether this pair is real or fake.

The model computes a dot product between the embedding of “pasta” and the embedding of “recipe.” A large positive dot product means the model thinks they belong together. A negative or small dot product means they don’t. The model is also shown fake pairs, like “pasta” and “elephant,” sampled at random. It must learn to give high scores to real pairs and low scores to fake ones.

{
  "type": "bar",
  "title": "Dot product similarity between word embeddings",
  "caption": "Illustrative similarity scores. High scores indicate words that often appear together.",
  "data": [
    {
      "label": "pasta - recipe",
      "value": 0.8
    },
    {
      "label": "pasta - sauce",
      "value": 0.75
    },
    {
      "label": "pasta - car",
      "value": 0.1
    },
    {
      "label": "pasta - quantum",
      "value": -0.2
    }
  ]
}

Over time, the embeddings shift so that words that often appear together get vectors with large dot products. Words that never appear together get vectors that point in different directions. Because the model sees millions of real and fake pairs, it ends up encoding subtle semantic relationships. “Pasta” and “spaghetti” both appear with “recipe,” so they both get pushed toward the same region of the space.

Researchers later proved that this process is equivalent to factorizing a matrix of pointwise mutual information (PMI) between words and contexts. PMI measures how much more often two words co-occur than you would expect by chance. The model is essentially learning a low-dimensional approximation of that PMI matrix. That is why the numbers capture meaning. They are a compressed representation of statistical association.

In practice, the model never builds the full PMI matrix. It learns the vectors directly from the data stream, which scales to billions of words. The embedding dimension, typically a few hundred, is a hyperparameter that controls how much information can be packed into each vector. More dimensions can capture finer distinctions, but they also cost more memory and computation.

How are embeddings used beyond single words?

A chatbot needs to understand whole sentences, not just individual words. After the token embeddings are looked up, the transformer layers from Part 3 mix them together using attention. The final output is a sequence of vectors, one per token, each now carrying information about the entire sentence.

To get a single vector for the whole sentence, a common trick is to take the vector corresponding to a special token, like [CLS], or to average all token vectors. This sentence embedding can then be compared to other sentence embeddings using cosine similarity. If two sentences have a high cosine similarity, the model considers them semantically close.

This is how a chatbot knows that “how do I make pasta” and “pasta cooking instructions” are asking the same thing. The query is turned into a sentence embedding, and the chatbot compares it to embeddings of candidate answers or documents. The one with the highest similarity is returned.

Modern sentence embedding models are often trained with contrastive learning. They take a sentence, create two slightly different versions (like by dropping words or using a paraphrase), and force the model to make those two versions have nearly identical embeddings. At the same time, they push embeddings of unrelated sentences apart. This makes the embedding space more uniform and reliable for similarity comparisons.

This same idea powers recommendation systems. A user’s interaction history can be averaged into a user embedding. Items get their own embeddings. The dot product between user and item embeddings predicts how much the user will like the item. Behind every “you might also like” is a nearest-neighbor search in embedding space.

What does this mean for search, recommendations, and memory?

When you type a query into a chatbot, it doesn’t do a keyword match. It converts your query into an embedding and finds the stored passages whose embeddings are closest. This is semantic search. It works even when the words don’t overlap at all. “Inexpensive lodging near the beach” and “budget hotel by the sea” will map to nearby points, so the system retrieves the same results.

This is also how a chatbot can “remember” facts from earlier in a conversation. Some architectures store the conversation history as a set of embeddings and retrieve relevant parts when you ask a follow-up question. The model doesn’t have a photographic memory. It has a vector database that finds sentences with similar meaning to your new query.

Recommendation systems use the same principle. Every song, movie, or product gets an embedding learned from user behavior. The system finds items whose embeddings are close to yours. The math is the same as word similarity, just applied to different kinds of entities.

The takeaway is that embeddings are the universal adapter. They turn messy, discrete things, words, sentences, users, items, into a clean, continuous geometry where distance equals similarity. Every modern AI system that deals with language or personalization relies on this trick.

Quick Reference

Frequently Asked Questions

Q: Do embeddings really capture meaning, or just word co-occurrence?

They capture co-occurrence patterns that happen to align with meaning. The model never accesses definitions or real-world knowledge. But for many practical tasks, the statistical shadow of meaning is enough. That is why “pasta” and “spaghetti” end up close, even though the model has never tasted either.

Q: Why are embeddings high-dimensional? Couldn’t we use just a few numbers?

A few dimensions can’t capture the many different ways words can be similar. “King” and “queen” are similar in royalty but differ in gender. “King” and “ruler” are similar in role but differ in formality. High-dimensional spaces give the model enough room to represent many independent axes of variation simultaneously.

Q: How do I choose the right embedding dimension?

It’s a tradeoff. Larger dimensions can capture more nuance but require more memory and slow down computation. For word embeddings, 300 is a classic sweet spot. For sentence embeddings from transformers, 768 or 1024 is common. Start with what the pretrained model provides and only reduce if you have tight latency constraints.

Q: Can I use embeddings for languages other than English?

Yes. The same principles apply to any language. Multilingual models train on many languages at once and learn a shared embedding space where “pasta” in English and “pâtes” in French end up close if they appear in similar translated contexts. This enables cross-lingual search without explicit translation.

Q: Are embeddings the same as the hidden states inside a transformer?

Not exactly. The token embeddings are the input to the transformer. The hidden states are the outputs after attention has mixed the token embeddings with context. Both are vectors, but hidden states are context-dependent: the vector for “bank” will be different in “river bank” versus “bank account.” Embeddings are context-independent lookup values.

Test yourself

You are building a chatbot that helps users find recipes. A user types “I want something warm and comforting for dinner.” The chatbot retrieves a recipe for chicken soup, which is correct. But when the user types “I need a cozy meal for tonight,” the chatbot retrieves a salad recipe. You suspect the embedding model is not capturing the similarity between “warm and comforting” and “cozy.” How would you diagnose and fix this?

Answer: First, check the cosine similarity between the sentence embeddings of the two queries. If it is low, the model sees them as unrelated. Next, examine the token embeddings for “warm,” “comforting,” and “cozy.” If “cozy” is far from the others, the pretraining data likely lacked examples where “cozy” appeared in food contexts. To fix this, you can fine-tune the sentence embedding model on a small dataset of paraphrased recipe queries, using contrastive learning to pull “cozy meal” and “warm comforting dinner” together. Alternatively, you can augment the retrieval index with synonyms or use a cross-encoder reranker that compares the query and candidate directly, which is more accurate but slower. The root cause is that the generic embedding space doesn’t specialize in your domain’s phrasing, and fine-tuning or reranking bridges that gap.

If you want this kind of breakdown every week, how real AI systems actually work under the hood, from tokens to attention to embeddings and beyond, subscribe to Internals Decoded at internalsdecoded.com. The next episode will show how these embeddings are used to build a model’s memory, allowing it to recall facts from its training data without storing every sentence.

Sources

• Efficient Estimation of Word Representations in Vector Space (Word2Vec)
• Distributed Representations of Words and Phrases and their Compositionality (Negative Sampling)
• GloVe: Global Vectors for Word Representation
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Attention Is All You Need (Transformer token embeddings)
• PyTorch nn.Embedding documentation

Originally published at Internals Decoded. AI internals, explained conversationally.