Embeddings: Meaning as Vectors

Introduction

After tokenization, your text is a list of integer IDs — and an ID is meaningless. Token 9059 ("cat") isn't "close to" token 9060 in any real sense; the numbers are just arbitrary labels. So how does a model do anything with meaning?

The answer is the single most important idea in modern AI: embeddings. Each token is turned into a vector — a list of numbers — positioned in space so that things with similar meaning sit close together. That turns meaning into geometry, where "related" becomes "nearby" and you can do math on concepts.

This one idea is the bedrock of semantic search, RAG, recommendations, and classification — and it's how the model itself begins to "understand" your tokens.

You'll learn:

Why arbitrary token IDs can't capture meaning — and how vectors fix it
The meaning space: where distance = similarity
The famous king − man + woman ≈ queen property
Token embeddings (inside the model) vs text embeddings (for search)
How to measure similarity, and see it in code

From Arbitrary IDs to Meaningful Vectors

A token ID is like a seat number in a stadium: "seat 9059" tells you nothing about who's sitting there. We need a representation where the numbers themselves carry meaning.

An embedding replaces each token ID with a dense vector — a list of, say, 768 / 1,536 / 3,072 numbers (the dimensions). Crucially, these numbers are learned from data, not assigned by hand. During training, the model nudges each token's vector until tokens that appear in similar contexts end up with similar vectors.

"cat"     → [ 0.18, -0.04,  0.91, ... ]   (e.g. 1536 numbers)
"kitten"  → [ 0.21, -0.02,  0.88, ... ]   ← very close to "cat"
"democracy" → [-0.77,  0.40, -0.12, ... ] ← far from both

The vector is the meaning, encoded as a point in a high-dimensional space.

The Meaning Space

Picture all those vectors as points in a space. The defining property: distance encodes similarity. Words with related meaning cluster together; unrelated ones are far apart.

We can't visualize 1,536 dimensions, so imagine it squashed down to a 2-D map: animals land in one neighborhood, countries in another, royalty in another. "cat" sits beside "kitten" and "dog"; "France" sits beside "Japan"; "democracy" is off in its own region entirely.

This is what people mean by an embedding space (or latent space): a learned map of meaning where geometry — how close two points are — answers the question "how related are these two things?"

A 2-D map of an embedding space. Related words form clusters: an 'animals' cluster (cat, kitten, dog), a 'countries' cluster (France, Japan, Brazil), and a 'royalty' cluster (king, queen, man, woman). Two parallel arrows illustrate the analogy: an arrow from 'man' to 'woman' is parallel to an arrow from 'king' to 'queen', showing that king − man + woman lands near queen. A caption notes that similar meanings sit close together and that relationships are consistent directions.

Doing Math on Meaning

Here's the property that made embeddings famous. Because relationships become consistent directions in the space, you can do arithmetic on meaning:

vector("king") − vector("man") + vector("woman")  ≈  vector("queen")

Subtracting "man" and adding "woman" moves you along the "gender" direction — turning king into queen. The same trick gives Paris − France + Japan ≈ Tokyo (the "capital-of" direction). This was the headline discovery of word2vec, and it proved embeddings capture relationships, not just clusters.

Modern models go further with contextual embeddings: the same word gets a different vector depending on its sentence. "bank" in "river bank" and "bank account" land in different places — because meaning depends on context. (Older word2vec gave every word one fixed vector; today's models don't.)

Two Flavors: Token Embeddings vs Text Embeddings

You'll meet embeddings in two closely related roles:

Token embeddings (inside the model). The model's very first layer — the embedding layer — converts each token ID into a vector. These vectors are what actually flow into the transformer (next lesson). This is how the model turns symbols into something it can reason over.
Text embeddings (for your applications). You can also embed an entire sentence or document into a single vector that captures its overall meaning. Compare two such vectors and you know how semantically similar two texts are — even if they share no words.

That second flavor is the engine of semantic search and RAG: embed a question, embed your documents, and retrieve the documents whose vectors are closest to the question's. We build exactly that in the RAG container — this lesson is the foundation it rests on.

Measuring Similarity (Lightly)

To turn "close together" into a number, the standard tool is cosine similarity — it measures the angle between two vectors:

≈ 1 → pointing the same way → very similar meaning
≈ 0 → perpendicular → unrelated
< 0 → opposite directions → opposite meaning

It cares about direction, not length, which is what we want when comparing meaning. We'll go deep on cosine vs dot vs Euclidean distance in the RAG container — here, just hold the intuition: smaller angle = more similar.

See It in Code

Let's confirm the intuition. We embed three words with an embedding model and compare them — "cat" should be close to "kitten" and far from "democracy."

import numpy as np
from openai import OpenAI
client = OpenAI()

def embed(texts):
    r = client.embeddings.create(model="text-embedding-3-small", input=texts)
    return np.array([d.embedding for d in r.data])

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

v = embed(["cat", "kitten", "democracy"])
print("cat ~ kitten   ", round(cosine(v[0], v[1]), 3))   # ~0.82  (high — similar)
print("cat ~ democracy", round(cosine(v[0], v[2]), 3))   # ~0.09  (low  — unrelated)

The model has never been told that cats and kittens are related — it learned that purely from how the words are used across its training data, and baked it into the geometry. (text-embedding-3-small is OpenAI's embedding model; open-source options like sentence-transformers work the same way and run locally — more on choosing one in the RAG container.)

It's Meaning, Not Spelling (the real magic)

Here's the moment embeddings click emotionally. Watch what happens with words that mean the same thing but share almost no letters:

pairs = [("doctor", "physician"), ("car", "automobile"),
         ("happy", "joyful"),  ("cat", "banana")]

for a, b in pairs:
    va, vb = embed([a, b])
    print(f"{a:7} ~ {b:11} {cosine(va, vb):.2f}")

# doctor  ~ physician   0.71
# car     ~ automobile  0.78
# happy   ~ joyful      0.69
# cat     ~ banana      0.11   (unrelated -> low)

"car" and "automobile" don't share a single letter — yet they're neighbors in the space, because the model learned they're used the same way. That's the whole magic: embeddings match meaning, not spelling.

This is exactly why semantic search beats keyword search. A search for "feline" can retrieve a document about "cats" even though the word "cat" never appears — because their vectors are close. Embed the query, embed the documents, return the nearest vectors: you've matched by meaning. That single trick is the engine of RAG, which you'll build later — and now you can see why it was inevitable.

🧪 Try It Yourself: The Embedding Map

Embeddings turn words into points in space where meaning = location. Click any word in the map below and watch its nearest neighbours light up:

Click cat → kitten, dog, puppy (animals cluster together).
Click king → queen, prince (royalty sits in its own region).
Notice fruit, animals, royalty, and verbs each form their own neighbourhood — that's what an embedding model learns: similar meanings end up close.

Embedding map — click a word, see its nearest neighbours.

Mental-Model Corrections

Embeddings aren't the model's full "understanding." They're the input representation — the starting point. Real understanding builds up as those vectors pass through the transformer's layers (next lesson).
Similar ≠ identical, and ≠ true. Close vectors mean semantically related, not factually equivalent. "Paris" and "London" are near each other (both capitals) — nearness is about relatedness, not sameness.
Embedding spaces are model-specific. A vector from one model is gibberish to another. Never mix embeddings from different models — always embed everything you'll compare with the same model.
More dimensions ≠ always better. Bigger vectors capture more nuance but cost more to store and compare; it's a tradeoff (covered in RAG).

Key Takeaways

An embedding turns a token (or whole text) into a vector in a learned space where distance = similarity — meaning becomes geometry.
Relationships become consistent directions, enabling analogies like king − man + woman ≈ queen.
Modern embeddings are contextual — the same word gets different vectors in different contexts.
Token embeddings feed the model; text embeddings power semantic search & RAG.
Compare vectors with cosine similarity (angle): ~1 similar, ~0 unrelated. Always compare embeddings from the same model.

Next: tokens are now meaningful vectors — but they're still considered in isolation. The breakthrough that lets a model weigh how words relate in context is attention, the heart of the Transformer — which we meet next.