From Words to Vectors (Depth)

Back in Embeddings: Meaning as Vectors you learned what an embedding is — a point in space where distance = similarity, so doctor lands next to physician. That lesson showed the what. This one opens the box on the how — and it's the engine under every RAG system, semantic search, and recommendation you'll build for the rest of this container.

Two questions most courses never answer:

1. How does a model turn a whole sentence — many tokens — into one fixed-size vector?
2. The deeper one: nobody hand-labels the axes, so how did the model learn a space where similar meanings end up close?

The answer to #2 — contrastive learning — is one of the most important ideas in applied AI, and once you see it, embeddings stop being magic.

In this lesson you'll learn:

• How an embedding model turns text into one vector (encoder + pooling)
• How the space is learned — contrastive training, bi-encoders, in-batch negatives
• What the dimensions mean (and how many you need)
• Normalization and instruction/asymmetric embeddings (the query: / passage: trick)
• A quick map of the 2026 embedding model landscape

Recall from Section 2: a transformer turns a sequence into a context-aware vector per token. But a search index needs one vector for the whole text. An embedding model adds exactly one step on top of the encoder: a pooling layer that collapses all the token vectors into a single fixed-size vector.

Two common pooling methods:

• Mean pooling (the default) — average all the token vectors element-wise. Simple and robust; what most sentence-embedding models use.
• CLS / last-token pooling — take the vector at one special position (a [CLS] token, or the final token) that the model learned to use as a summary.

So a modern text-embedding model — a bi-encoder (a.k.a. dual-encoder or sentence-transformer) — is just a transformer encoder + a pooling layer. "Embed this sentence" = run it through the encoder, pool the token vectors, get one vector of, say, 768 numbers.

Pooling gives you a vector — but why does that vector land near other related ones? Nobody programs an "is-it-medical" axis. The geometry is learned, and the method is contrastive learning.

The model is shown pairs that should be close (a question and its correct answer; a sentence and its paraphrase; a title and its article) and pairs that should be far, and trained to pull the close pairs together and push the far pairs apart in vector space.

Repeat that over hundreds of millions of pairs and the space self-organizes: anything that tends to co-occur or mean the same thing drifts together; unrelated things drift apart. The result is the geometry you used last lesson — learned from examples, not labeled by hand.

This is the idea. Everything else is detail.

Here's the machinery, with just enough depth to reason about it:

The bi-encoder. The same encoder (shared weights) embeds both items in a pair — the query and the passage go through identical towers. That's why query and document vectors live in the same space and can be compared directly.

The loss. For a training example (anchor, positive), the objective maximizes the similarity of anchor↔positive while minimizing anchor↔negatives. Pull the right pair together; push the wrong pairs apart.

The clever efficiency trick — in-batch negatives. Where do negatives come from? You don't need to hand-pick them. In a training batch of N (query, passage) pairs, each query's own passage is its positive — and the other N−1 passages in the batch are negatives for free. One batch gives you N positives and N×(N−1) negatives at almost no extra cost.

A direct consequence you'll see referenced everywhere: bigger batches make better embeddings — more passages per batch means more (and harder) negatives to contrast against. It's why embedding training runs use very large batch sizes.

Hard negatives push it further: passages that look relevant but aren't (e.g., the right topic, wrong answer) are deliberately added so the model learns fine distinctions, not just "medical vs. cooking." Modern models (E5, BGE, GTE, Nomic) are trained on a mix of in-batch and mined hard negatives — which is exactly why they're so good at retrieval.

Each embedding is a list of numbers — its dimensions. A few honest clarifications:

• The dimensions are learned features, not human-interpretable. There is no single "sentiment" or "is-an-animal" axis you can point to. Meaning is distributed across all of them at once — that distributed encoding is what lets a few hundred numbers capture so much nuance.
• More dimensions = more capacity, more cost. Bigger vectors can encode finer distinctions, but they cost more to store and are slower to compare (every similarity is a dot product over all dimensions). Typical sizes in 2026:

You don't always have to pick the biggest — and you don't have to commit forever: Matryoshka embeddings let you truncate a big vector to a smaller one with minimal quality loss (its own lesson — Matryoshka & Quantized Embeddings). For now: more dims ≠ automatically better; it's a quality-vs-cost dial.

Two practical depths that bite people in production:

1. Normalization. Most embedding models output (or you should normalize to) unit-length vectors — every vector sits on a sphere, so only its direction carries meaning, not its length. The payoff: for unit vectors, cosine similarity equals the dot product, which is faster and is what vector databases optimize for. (We measure similarity properly in the next lesson — for now, just normalize.)

2. Asymmetric / instruction embeddings — the query: / passage: trick. Here's a subtlety that silently wrecks naive RAG: a question and its answer are phrased completely differently ("how do I reset my password?" vs "Click 'Forgot password' on the login page."). Modern models (E5, BGE, Gemini, Nomic, Qwen3) are trained to handle this by having you prefix the input with its role or a task instruction:

• embed the question as "query: how do I reset my password?"
• embed the document as "passage: Click 'Forgot password'..."

Telling the model which side a text is on measurably improves retrieval — and omitting the prefix can quietly drop quality by several points. Always check your embedding model's docs for the exact instruction format, and use it consistently for queries vs. documents.

You'll choose among these next lesson; here's the lay of the land so the names aren't a mystery. They're all ranked on MTEB (the Massive Text Embedding Benchmark — the standard leaderboard, read it as critically as any benchmark):

• Gemini Embedding — tops MTEB in 2026; multimodal (text/image/audio/PDF in one shared 3072-d space) with a task-type parameter.
• OpenAI text-embedding-3-small / -large — the safe defaults (1536 / 3072 dims, Matryoshka-truncatable). -small at ~$0.02/1M tokens covers ~90% of apps.
• Voyage 3 — reach for it when retrieval quality is the bottleneck.
• BGE-M3 / Qwen3 / GTE / E5 (open-weight) — strong, free, self-hostable; BGE-M3 + a BGE reranker is a very common production default.
• Nomic-embed-text — the most-pulled embedding model on Ollama; ~137M params, runs on a laptop CPU, 8K context.

Don't memorize the leaderboard (it moves monthly). Know the families, and pick by task, language, dimension budget, and cost — exactly the workflow of the next lesson.

Produce real embeddings two ways — a local open model (note the role prefixes) and an API (note the Matryoshka dimensions):

That's the whole interface you'll use across this container: text in → one vector out. Everything in RAG is built on comparing those vectors.

Feel the asymmetry. Embed a question and its real answer, which share almost no words, two ways:

1. Naive: embed both as plain text (no prefixes) and take the cosine similarity.
2. Correct: embed the question as "query: …" and the answer as "passage: …" (for an E5/BGE-style model) and compare again.

Predict, then check: which scores the pair as more similar? → The prefixed version, usually by a clear margin — because you told the model these are two sides of a retrieval pair, not two unrelated sentences. Now flip the experiment: embed two unrelated sentences with the prefixes — does the score stay low? (It should.) That single habit — right prefix for queries vs. documents — is one of the cheapest retrieval wins there is.

• "The dimensions are meaningful axes." No — they're learned, distributed features; no single dimension is "the medical axis." Meaning lives across all of them.
• "The model was told what's similar." No — it learned the space from contrastive pairs (pull positives, push negatives). The geometry is emergent, not labeled.
• "An embedding is the model's full understanding." It's a compressed input representation of meaning — powerful, but a lossy summary, not reasoning.
• "Bigger vectors are always better." More dims = more capacity and more cost; the best model is task- and budget-dependent (and Matryoshka lets you shrink).
• "Queries and documents embed the same way." Often no — instruction-tuned models want role prefixes (query: / passage:); skipping them quietly hurts retrieval.
• "Any two models' vectors are comparable." Never — embeddings are model-specific; always embed everything you'll compare with the same model.

• An embedding model is an encoder + pooling: it turns a text's many token vectors into one fixed-size vector (mean pooling, usually).
• The space is learned by contrastive training — pull semantically-similar (positive) pairs together, push negatives apart — using a bi-encoder and in-batch negatives (which is why big batches help).
• Dimensions are learned, distributed features (not interpretable axes); more dims = more capacity and more cost (384 → 4096; Matryoshka can shrink them).
• Normalize to unit vectors (so cosine = dot product), and use role prefixes (query: / passage:) for instruction-tuned models — omitting them silently hurts retrieval.
• Know the 2026 families (Gemini, OpenAI 3-small/large, Voyage, BGE/Qwen/E5, Nomic) and compare on MTEB — but choose by task + cost (next lesson).