Mean Pooling vs CLS: Watch Transformers.js Turn a Sentence Into an Embedding in Node

Book: AI That Reads
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Somewhere in your RAG code there is a line like const vec = await embed(text). A list of 384 floats comes back, you store it, and you never look inside again.

That function hides a choice. In Transformers.js, the feature-extraction pipeline has two options that decide what those floats mean, and both default to "do nothing": pooling defaults to "none" and normalize defaults to false, per the pipeline API docs. Call it with no options on "This is a simple test." and the docs show you get a tensor of shape [1, 8, 384]. You get eight vectors back, one per token.

Every embedding library makes this choice for you somewhere. When the indexing job and the query path make it differently, nothing throws. Retrieval just gets quietly worse. Open the box in Node one step at a time, then break it on purpose.

What the model actually returns

Install the library (version 4.3.1 is the current release on npm, and in Node it runs ONNX models through ONNX Runtime). Run the script with tsx. The model downloads from the Hugging Face Hub on first run. No API key.

npm i @huggingface/transformers
npx tsx embed.mts

The model is an ONNX export of all-MiniLM-L6-v2, the same one the Transformers.js docs use. Its model card says it maps text to a 384-dimensional space, uses mean pooling, and truncates input longer than 256 word pieces.

// embed.mts
import {
  AutoModel,
  AutoTokenizer,
  type Tensor,
} from "@huggingface/transformers";

const ID = "onnx-community/all-MiniLM-L6-v2-ONNX";
const tokenizer = await AutoTokenizer.from_pretrained(ID);
const model = await AutoModel.from_pretrained(ID, {
  dtype: "fp32",
});

const query = "I can't log in. How do I change my password?";
const docs = [
  "Reset your password from the account settings page.",
  "Our office is closed on public holidays.",
  "Two-factor codes expire after 30 seconds.",
  "Passwords must be at least 12 characters long.",
  "Contact billing to update your credit card.",
];
const all = [query, ...docs];

console.log(tokenizer.tokenize(query));
const enc = await tokenizer(all, {
  padding: true,
  truncation: true,
});
console.log(enc.input_ids.tolist()[0]);
console.log(enc.attention_mask.tolist()[1]);

const out = (await model(enc)) as Record<string, Tensor>;
const hidden = out.last_hidden_state ?? out.token_embeddings;
console.log(hidden.dims); // [batch, tokens, 384]

Three things to read in that output.

tokenize shows the WordPiece split. Words outside the vocabulary break into sub-word pieces, so the token count is not the word count.

input_ids has more entries than tokenize printed. The tokenizer adds special tokens at both ends by default (add_special_tokens defaults to true). The one at position 0 is [CLS]. Remember that position.

attention_mask is a row of 1s followed by 0s for any sentence shorter than the longest one in the batch. With padding: true, every row is stretched to the same length, and the mask marks which positions are real text. The tokenizer docs describe it as "attended to (1) versus padded (0)".

The model output is [6, N, 384]: six texts, N positions each, 384 numbers per position. The fallback to token_embeddings mirrors what the pipeline itself does in its source: it reads last_hidden_state, then logits, then token_embeddings, depending on how the ONNX file was exported.

Pooling, written out by hand

Pooling collapses N token vectors into one. The two common choices are mean pooling (average the real tokens) and CLS pooling (take position 0 and ignore the rest). Both poolings and L2 normalization fit in a few lines of plain TypeScript. Append this block, and every block after it, to embed.mts and rerun the script.

const H = hidden.tolist() as number[][][];
const M = (enc.attention_mask.tolist() as bigint[][])
  .map((row) => row.map(Number));

function meanPool(tokens: number[][], mask: number[]) {
  const sum = new Array(tokens[0].length).fill(0);
  let count = 0;
  tokens.forEach((vec, t) => {
    if (mask[t] === 0) return; // skip padding
    count++;
    vec.forEach((x, d) => (sum[d] += x));
  });
  return sum.map((x) => x / count);
}

const clsPool = (tokens: number[][]) => tokens[0];

function l2(v: number[]) {
  const norm = Math.hypot(...v);
  return v.map((x) => x / norm);
}

const dot = (a: number[], b: number[]) =>
  a.reduce((s, x, i) => s + x * b[i], 0);
const cosine = (a: number[], b: number[]) =>
  dot(a, b) / (Math.hypot(...a) * Math.hypot(...b));

The if (mask[t] === 0) return line is the whole reason the mask exists. Padding positions still get hidden states out of the model. Average them in, and a short sentence's vector depends on how long its neighbours in the batch were. The sentence-transformers model card carries a comment on its own mean_pooling helper for this reason: "Take attention mask into account for correct averaging."

Check your version against the library's. The pipeline does the same three steps internally: it tokenizes with padding: true, truncation: true, passes the attention mask into mean_pooling for "mean", slices position 0 for "cls", and calls normalize(2, -1) when you ask for normalization (source).

import { pipeline } from "@huggingface/transformers";

const extractor = await pipeline("feature-extraction", ID, {
  dtype: "fp32",
});
const ref = (
  await extractor(all, { pooling: "mean", normalize: true })
).tolist() as number[][];

const mine = all.map((_, i) => l2(meanPool(H[i], M[i])));
const diffs = mine.flatMap((v, i) =>
  v.map((x, d) => Math.abs(x - ref[i][d])),
);
console.log("max abs difference:", Math.max(...diffs));

You are looking for a number at float-rounding scale. If it is large, your pooling does not match the library's, and that is worth finding out before you index anything. Try deleting the mask check and run it again: the drift on the shorter sentences is the padding leaking in.

Normalization changes dot product, not cosine

Cosine similarity divides by both vector lengths, so normalizing first does not change a cosine score. Print cosine(a, b) next to cosine(l2(a), l2(b)) for any two pooled vectors and they agree up to rounding.

What normalization changes is the dot product. On unit-length vectors, dot product and cosine are the same number. On raw vectors, dot product also rewards length, so a document whose pooled vector happens to be longer can climb the ranking without being more relevant.

That matters because vector stores often ask you to pick the operator. The pgvector README lists <#> for negative inner product and <=> for cosine distance, and it recommends: "If vectors are normalized to length 1 (like OpenAI embeddings), use inner product for best performance." Follow that advice with normalize: false vectors and you have changed your ranking function without changing a line of retrieval code.

Run the same query four ways

The experiment reuses one forward pass over the same six strings and turns the hidden states into a ranking four different ways.

function rank(name: string, q: number[], d: number[][]) {
  console.log(`\n${name}`);
  d.map((v, i) => ({ i, s: dot(q, v) }))
    .sort((a, b) => b.s - a.s)
    .forEach(({ i, s }) =>
      console.log(`  ${s.toFixed(4)}  ${docs[i]}`));
}

const mean = all.map((_, i) => meanPool(H[i], M[i]));
const cls = all.map((_, i) => clsPool(H[i]));
const meanL2 = mean.map(l2);
const clsL2 = cls.map(l2);

rank("A. mean + L2 (model card)", meanL2[0], meanL2.slice(1));
rank("B. cls + L2", clsL2[0], clsL2.slice(1));
rank("C. mean, raw dot", mean[0], mean.slice(1));
rank("D. cls query vs mean index", clsL2[0], meanL2.slice(1));

Run A is the configuration the model card describes, so treat it as the baseline. Compare the others to it.

Run B uses a different pooling on both sides. The scores move, because the sentence vectors are different vectors. Whether the top result moves depends on your sentences. The model card pools with mean, so position 0 was not the pooling target during fine-tuning. Expect it to be the weaker signal.

Run C keeps mean pooling but skips normalization and ranks by raw dot product. Compare the order to A. Any reshuffle you see comes from vector length alone.

Run D is the one that hurts in production: the index was built with mean pooling, the query path uses CLS. Both vectors are 384 floats with unit length. Your database accepts them, the query returns five rows, and the scores look like scores. Nothing in the pipeline tells you the two sides were produced by different functions.

Swap in your own query and a few dozen of your own documents, and watch each step change the output.

The mismatch is one copy-paste away

BGE models are trained the other way. The sentence-transformers pooling config shipped in BAAI/bge-small-en-v1.5 sets "pooling_mode_cls_token": true and "pooling_mode_mean_tokens": false, and the model card's Transformers example pools with model_output[0][:, 0]. The BGE tutorial says BGE "is trained to use the last hidden state of [CLS] as the sentence embedding" and warns that "If you use mean pooling, there will be a significant decrease in performance."

The Transformers.js-ready port of the same model, Xenova/bge-small-en-v1.5, shows { pooling: 'mean', normalize: true } in its usage examples. The example is consistent with itself, because indexing and query both use mean. But copy it into a Node indexer, query the same collection from a Python service that loads BAAI/bge-small-en-v1.5 through sentence-transformers (whose modules.json wires in the CLS pooling config above), and you have Run D in production.

Cloudflare Workers AI documents the same split for its hosted bge-base-en-v1.5. Its docs say the pooling parameter defaults to mean "in order for this to not be a breaking change," recommend cls for accuracy, and state plainly that "embeddings created with cls pooling are not compatible with embeddings generated with mean pooling."

That last sentence is the rule. Pooling is part of the model, as far as your index is concerned.

Pin the embedding config next to the vectors

Treat the settings that produce a vector as one value, and store that value with the index.

export const EMBED = {
  model: "onnx-community/all-MiniLM-L6-v2-ONNX",
  dtype: "fp32",
  pooling: "mean",
  normalize: true,
  queryPrefix: "",
} as const;

export const EMBED_KEY = JSON.stringify(EMBED);

export function assertSameSpace(indexKey: string) {
  if (indexKey !== EMBED_KEY) {
    throw new Error(
      `index built with ${indexKey}, querying with ${EMBED_KEY}`,
    );
  }
}

Write EMBED_KEY into a metadata row when you build the index. Call assertSameSpace when the query service starts. If a second service in another language reads the same index, give it the same key and the same check.

dtype is in there for a reason: Transformers.js lets you load quantized variants (q8, q4, fp16 and more, per the dtypes guide), and a quantized model is a different set of weights, so pin it like the rest. queryPrefix is in there because the BGE model card suggests an instruction prefix on short retrieval queries and not on passages, which is one more asymmetric setting that is easy to forget on one side.

One more trap if you move to a decoder-style embedding model that uses last_token or eos pooling: the pipeline takes the final position along the sequence axis without consulting the attention mask (source). Check which side your tokenizer pads on before you batch, or the "last token" of a short sentence may be padding.

If this was useful

This post is the inside of one function. AI That Reads is the book around it: embeddings, chunking, vector search and grounding answers in your own docs, written in TypeScript for Node. If you liked seeing where the 384 floats come from, the book keeps going from there, into how to store them, search them, and tell when retrieval is quietly failing.