Matryoshka & Quantized Embeddings (Cost/Scale)

Introduction

Embeddings are cheap to create but can be brutally expensive to store and search at scale. Do the math: one million documents × a 1536-dimension float32 vector = ~6 GB of vectors. A billion documents → ~6 TB — and vector search is memory-bound, so that mostly has to live in RAM. At scale, embedding storage — not the API calls — becomes the dominant cost.

Two techniques shrink embeddings dramatically with surprisingly little quality loss, and they stack:

Matryoshka — use fewer dimensions (shorter vectors).
Quantization — use fewer bits per number (coarser vectors).

Together they can cut storage 10–256× — often the difference between "we can afford this" and "we can't." This is the lesson that makes embeddings scale.

You'll learn:

Matryoshka representation learning — and why a prefix of a vector still works
Quantization — int8 and binary, and the Hamming-distance trick
The coarse-to-fine retrieval pattern that keeps quality while slashing cost

Matryoshka: Use Fewer Dimensions (Almost for Free)

Normally, shrinking a vector means re-embedding with a smaller model. Matryoshka Representation Learning (MRL) does something cleverer: it trains the model so that the first N dimensions of the full vector are themselves a valid, semantically-rich embedding. Like a Russian nesting doll, a smaller embedding lives inside the big one.

So you can truncate a 1536-d vector to 512, 256, or even 128 dimensions — just keep the first N — and it still works, with graceful quality loss. Two facts that surprise people:

Truncating to 128 dims alone cuts storage by ~57% — for free, no re-embedding.
A truncated MRL embedding usually beats a model that's natively that small — you get the big model's quality, shrunk.

One critical gotcha: after you slice a vector, you must re-normalize it (the prefix isn't unit-length on its own). OpenAI's text-embedding-3 models, Nomic, Voyage, and Gemini all support this — via a dimensions parameter or manual truncation.

An infographic titled 'Shrink Embeddings 10 to 256 times — Matryoshka + Quantization' with two panels. The indigo MATRYOSHKA panel (fewer dimensions) shows a vector as a segmented bar marked 128, 256, 512, 1536, with the note that you can cut at any point because every prefix of the vector is a usable embedding; slicing a 1536-d vector down to 512, 256, or 128 still works with graceful quality loss, to 128 dims is about 57 percent less storage, and it beats a natively-small model; OpenAI exposes this via a dimensions parameter. The amber QUANTIZATION panel (fewer bits per number) lists float32 (32 bits, 4 bytes per dim, baseline), int8 (8 bits, 4 times smaller, tiny quality loss), and binary (1 bit, 32 times smaller, compared with Hamming distance). A callout: combine them — Matryoshka (up to 8x) times binary (32x) equals up to 256 times smaller; a 1024-d float32 vector (4 KB) becomes a 128-d binary vector (16 bytes), and at a billion vectors 4 TB becomes 16 GB. A green callout describes the production coarse-to-fine pattern: retrieve candidates fast with cheap (binary or truncated) vectors over everything, then rescore the top-k with full-precision vectors for near-full quality at a fraction of the cost. A bottom banner: embeddings are cheap to make, expensive to store at scale — shrink them, quality plateaus, cost drops.

See It: Truncating an Embedding

from openai import OpenAI
import numpy as np
client = OpenAI()

# Native Matryoshka: just ask the API for fewer dimensions
r = client.embeddings.create(model="text-embedding-3-large",
                             input="how do I reset my password?", dimensions=256)
print(len(r.data[0].embedding))   # 256  (the full model is 3072) -> 12x smaller, ~full quality

# Manual truncation of ANY MRL vector: slice, then RE-NORMALIZE (don't skip this!)
def truncate(vec, d):
    v = np.array(vec)[:d]          # keep the first d dimensions
    return v / np.linalg.norm(v)   # re-normalize so cosine still works

Quantization: Use Fewer Bits per Number

Matryoshka shrinks how many numbers. Quantization shrinks how big each number is — its precision. Embedding values are normally float32 (32 bits = 4 bytes each). You rarely need that much precision:

int8 (scalar quantization) — map each value to an 8-bit integer → 4× smaller, with tiny quality loss. A near-free win; do it almost always.
binary quantization — collapse each dimension to a single bit (is it positive?) → 32× smaller. You compare binary vectors with Hamming distance (count differing bits — a blazing-fast bitwise op). The quality loss is bigger but, remarkably, binary vectors retain most of the retrieval signal — easily enough for a first pass.

Quantization is pure infrastructure savings: smaller index, less RAM, faster comparisons — for a quality cost you control.

from sentence_transformers.quantization import quantize_embeddings
import numpy as np

emb = np.random.randn(1000, 1024).astype(np.float32)    # your float32 vectors

int8   = quantize_embeddings(emb, precision="int8")     # 4x smaller
binary = quantize_embeddings(emb, precision="binary")   # 32x smaller (1 bit/dim)

print(emb.nbytes, int8.nbytes, binary.nbytes)
# 4,096,000  ->  1,024,000  ->  128,000   bytes   (4x, then 32x)

The Production Pattern: Coarse-to-Fine Rescoring

Here's how the best systems get near-full quality at a fraction of the cost — and it ties both techniques together. It's a two-stage, coarse-to-fine retrieval:

Coarse (cheap, over everything): search the whole corpus with tiny vectors — binary and/or Matryoshka-truncated — to fetch a generous candidate set (say, the top 100). This is fast and memory-light even over billions of vectors.
Fine (accurate, over a few): rescore just those ~100 candidates with the full-precision, full-dimension vectors, and keep the true top-k.

You only pay full-precision cost on the handful of candidates, not the whole index — so you keep ~the quality of full embeddings at a tiny fraction of the storage and compute. This "binary retrieval + rescoring" is one of the highest-leverage tricks in production vector search (the reranking section goes deeper on the rescore step).

Combine for Massive Savings

Because the two techniques are independent axes, they multiply:

Matryoshka truncation (up to ~8×) × binary quantization (32×) = up to ~256× smaller.

Concretely: a 1024-d float32 vector (4 KB) → a 128-d binary vector (16 bytes). At a billion vectors, that's ~4 TB → ~16 GB — from "needs a cluster" to "fits on one machine."

The dial is accuracy: more compression = a bit more quality loss. But thanks to coarse-to-fine rescoring, the first-pass loss barely matters — the full-precision rescore recovers it. As always: tune the compression on your eval, watching recall@k as you shrink. Most teams are shocked how far they can compress before quality moves.

🧪 Try It Yourself

Drag the dimensions in the widget below (it's the Matryoshka trade-off live):

Slide from 1536 → 256: watch storage/cost crater (to ~17%) while quality barely moves (~96%). That flat quality curve is why Matryoshka works.
Keep going to 64 dims: now quality finally drops off — there's a floor.

Then a sizing exercise: you have 50 million documents and a 3072-d float32 model (~600 GB of vectors — too much RAM). Combine the levers: truncate to 256-d (÷12) and quantize to int8 (÷4) → ÷48 → ~12.5 GB. Suddenly it fits on one box. Then add binary first-pass + rescoring if you need to go further. That's how embeddings scale.

Interactive: a compression studio. Over a corpus the user picks (1M, 50M, or 1B docs), starting from a 1024-d float32 baseline (4 KB/vector), two levers combine — a Matryoshka dimensions slider (1024 down to 64) and a quantization toggle (float32 / int8 4x / binary 32x) — plus a coarse-to-fine rescoring checkbox. As the user shrinks, per-vector bytes and total storage crater, a compression multiplier climbs toward ~256x, a quality bar plateaus near the top before dropping at a floor, and a 'fits on one box vs needs a cluster' verdict compares total storage against ~64 GB of RAM. Turning on rescoring shows the first-pass quality (e.g. binary at low dims) recovered to near-full because the finalists are rescored at full precision. The lesson made tangible: dimensions and bits are independent axes that multiply, storage is the dominant cost at scale, quality plateaus so you can compress aggressively for the first pass and rescore the finalists, and you tune the dial on your eval. Figures are an illustrative simulation.

Mental-Model Corrections

"A shorter vector means a worse, separate model." No — Matryoshka lets you truncate one big vector; the prefix is a valid embedding, and it usually beats a natively-small model.
"I can just slice the vector." Slice and re-normalize — the prefix isn't unit-length, so cosine breaks without it.
"Quantization wrecks quality." int8 is nearly free (~4×); binary (32×) loses more but keeps enough signal for a first pass — recovered by rescoring.
"Compression always hurts accuracy." With coarse-to-fine rescoring, first-pass loss barely matters — you rescore the finalists at full precision.
"Storage is a rounding error." At scale it's the dominant cost (vector search is memory-bound). Shrinking vectors is often the biggest cost lever in RAG.

Key Takeaways

Embeddings are cheap to make but expensive to store/search at scale — storage (RAM) is usually the dominant cost.
Matryoshka (MRL): the first N dims are a valid embedding, so you can truncate (1536→256→128) with graceful loss — and it beats a natively-small model. (Re-normalize after slicing! dimensions= on OpenAI.)
Quantization: lower the precision per number — int8 (4×, near-free) or binary (32×, compared with Hamming distance).
They stack: up to ~256× smaller (1024-d float32 → 128-d binary; 4 TB → 16 GB at a billion vectors).
The pattern: coarse-to-fine — retrieve candidates with cheap (binary/truncated) vectors, then rescore the top-k at full precision. Tune the compression on your eval.