Choosing & Fine-Tuning Embedding Models

Introduction

You know how embeddings work and how to compare them. Now the two practical decisions every RAG project faces: which embedding model do I use, and (more rarely) should I fine-tune one?

This matters more than it looks. The embedding model is the lens your entire knowledge base is seen through — if it can't tell your documents apart, no amount of clever prompting downstream will save you. And there's a switching cost: changing models later means re-embedding everything. So you want to choose well, once.

The two most common mistakes are opposite extremes: under-evaluating (picking whatever tops a leaderboard) and over-fine-tuning (training a custom model when a config change would've done it). This lesson fixes both.

In this lesson you'll learn:

The real selection criteria (beyond the MTEB rank)
Why you must read MTEB critically — and evaluate on your data
When to fine-tune (usually: don't) and when it's worth it
How to fine-tune — contrastive learning on your pairs, with hard negatives

The Selection Criteria (Beyond the Leaderboard Rank)

A model's MTEB rank is one input, not the decision. Weigh these against your use case:

Criterion	Ask yourself
Retrieval quality	How well does it rank relevant docs first — on a task like yours?
Dimensions	Bigger = more nuance and more storage/compute. 384–1024 is plenty for most; 1536–4096 for the frontier. (Matryoshka can shrink — L77.)
Context length	Can it embed your chunk size? Many models degrade past ~512 tokens — a mismatch silently truncates meaning.
Multilingual	Do your docs/queries span languages? If so, you need a multilingual model.
Domain fit	General web vs. legal/medical/code — does it know your vocabulary?
Open vs API	Privacy/cost/latency/control (the build-vs-buy call) — self-host an open model or call a hosted one?
License & cost	Commercial-use license? $/1M tokens (API) or GPU/storage (self-host)?

Notice none of these is "which is #1 on the leaderboard." The best model is the one that fits these constraints and wins on your data.

Read MTEB Critically

MTEB (the Massive Text Embedding Benchmark) is the standard leaderboard — retrieval, classification, clustering, reranking, similarity, multilingual. It's genuinely useful: it narrows dozens of models to a shortlist. But treat it like any benchmark (recall Reading Benchmarks Critically):

Its retrieval suite is general web — MS MARCO, TREC, Wikipedia-flavored data. If your corpus is legal contracts, scientific PDFs, or internal SaaS docs, MTEB retrieval scores are a directional signal at best.
Its passages are short — many under ~256 tokens. A model can top MTEB and then fall apart on your long, technical documents.
It's also susceptible to the usual contamination and saturation at the top.

So: use MTEB to shortlist 2–3 candidates that meet your criteria — then stop trusting it and run the only test that counts.

An infographic titled 'Choosing — and (Rarely) Fine-Tuning — an Embedding Model' with two columns. The indigo left column, CHOOSE a model: step A, shortlist on MTEB (the leaderboard) filtering by retrieval quality, dimensions/cost, context length, multilingual, license, and speed/API-vs-local; an arrow notes MTEB is general-web and short passages, not your domain; step B (starred, the decisive step), evaluate on YOUR data — take about 30 real query-to-known-relevant-doc pairs, embed with each candidate, measure recall@k, and the model that finds your docs wins, not the one that tops MTEB. The amber right column, FINE-TUNE? usually NO: step 1, try the cheaper wins first (better chunking, hybrid search, reranking, query/passage prefixes); step 2, only if domain jargon plus you have labeled pairs (legal/medical/code, and you can get or synthesize query-doc pairs); step 3, how — contrastive on your pairs via MultipleNegativesRankingLoss (in-batch negatives) plus hard negatives, the big lever. A red warning: switching models means re-embedding your whole corpus because vectors from different models aren't comparable, so choosing has a real switching cost. A bottom banner: most teams over-fine-tune and under-evaluate — eval on your data first, fine-tune last.

The Only Test That Matters: Evaluate on YOUR Data

Performance varies wildly by domain — a model that's #1 on MTEB can be #3 on your corpus. So before you commit, measure your shortlist on a held-out slice of your own data (exactly the model-selection discipline from Container 1, now for embeddings).

A lightweight retrieval eval is enough to choose:

Collect ~20–50 real (query → known-relevant document) pairs from your domain (support tickets → the KB article that answered them; questions → the doc with the answer).
Embed your whole doc set with each candidate model.
For each query, retrieve the top k and check whether the right doc is in there — recall@k ("did we find it?") and MRR/NDCG ("how high did it rank?").
The model with the best recall on your data wins.

This is an afternoon of work that routinely overturns the leaderboard pick. (Full, rigorous RAG evaluation — faithfulness, context precision/recall with RAGAS — comes later in this container; this lightweight version is enough to pick the embedder.)

When to Fine-Tune (Usually: Don't)

Fine-tuning a custom embedding model sounds like the serious-engineer move. For most teams it's the wrong first move — a strong general model plus the cheaper levers beats a hastily fine-tuned one. Try these before you fine-tune:

Better chunking (the single biggest RAG quality lever — a whole section ahead).
Hybrid search (add keyword/BM25 — catches exact terms embeddings miss).
Reranking (a cross-encoder reorders the top results — often a bigger win than a new embedder).
Instruction prefixes (query: / passage: — free quality you may be leaving on the table).

Fine-tune only when you've done the above and you have:

a genuine domain-vocabulary problem — legal, medical, or code, where a general model conflates terms that mean different things in your field; and
labeled (or synthesizable) (query, relevant-doc) pairs to train on.

The honest rule: a unique notion of similarity is the real reason to fine-tune. If "close" means something special in your domain that general models don't capture, fine-tuning teaches them — otherwise it's effort better spent on retrieval.

How to Fine-Tune: Contrastive Learning on Your Pairs

Here's the satisfying part: fine-tuning an embedding model is the exact contrastive learning from the Words to Vectors lesson — now on your data. You start from a strong pretrained model (never from scratch) and nudge its space so your relevant pairs land closer.

What you need: (anchor, positive) pairs — a query and a document that should match. Where they come from: existing signals (search clicks, support-ticket → resolving-article), or synthetic — have an LLM write a few plausible questions for each of your documents (a cheap, surprisingly effective way to bootstrap a training set).

The loss: MultipleNegativesRankingLoss — it needs only positive pairs and uses in-batch negatives automatically (recall: the other passages in the batch are free negatives, so bigger batches help).

The biggest lever — hard negatives. Positive-only pairs give marginal gains; adding hard negatives (passages that look relevant but aren't) drives substantial improvement, because they force the model to learn the fine distinctions your domain cares about. Mine them automatically; gains typically plateau around a few dozen negatives per query.

from sentence_transformers import (SentenceTransformer, losses,
    SentenceTransformerTrainer, SentenceTransformerTrainingArguments)
from datasets import Dataset

model = SentenceTransformer("BAAI/bge-base-en-v1.5")     # start from a strong PRETRAINED model

# (anchor, positive) pairs from YOUR data — e.g. (support question, the article that answers it)
train = Dataset.from_dict({
    "anchor":   ["how do I export my data?", "is there a student discount?"],
    "positive": ["Go to Settings -> Export to download a CSV of all your data.",
                 "Students get 50% off with a valid .edu email at checkout."],
})

loss = losses.MultipleNegativesRankingLoss(model)        # in-batch negatives = contrastive!

trainer = SentenceTransformerTrainer(
    model=model, train_dataset=train, loss=loss,
    args=SentenceTransformerTrainingArguments(
        output_dir="ft-embed", num_train_epochs=1,
        per_device_train_batch_size=32),                 # bigger batch -> more negatives -> better
)
trainer.train()
model.save_pretrained("ft-embed")   # now embed your corpus with the fine-tuned model

And the lightweight eval that should drive every embedding decision — choose by recall@k on your data, not by leaderboard rank:

import numpy as np

def recall_at_k(model, docs, queries, gold_idx, k=5):
    D = np.array(model.encode(docs, normalize_embeddings=True))         # embed the corpus once
    hits = 0
    for q, g in zip(queries, gold_idx):
        qv = model.encode(q, normalize_embeddings=True)
        topk = np.argsort(-(D @ qv))[:k]      # top-k by cosine (normalized -> dot)
        hits += int(g in topk)
    return hits / len(queries)

# run recall_at_k(...) for each candidate model on YOUR pairs; highest wins.

🧪 Try It Yourself

Make the call on a real project. Pick something you'd build (e.g. search over your company's engineering docs). Walk the decisions:

Criteria: which two matter most here — context length (long docs?), multilingual, domain fit, cost?
Shortlist: name 2–3 candidates (a frontier API model, a strong open model, a small/cheap one).
The decisive step: what would your ~30 (query → doc) eval pairs look like, and what's recall@5 measuring?
Fine-tune? Be honest — have you tried chunking + hybrid + reranking first? Do you actually have labeled pairs?

If your answer to #4 is "not yet" — that's the right answer for ~90% of projects. Eval first; fine-tune last.

Interactive: an embedding-model bench. Three candidates — a Frontier-API model (MTEB #1, 8k context, 3072-d), a Strong-Open model (BGE-M3, MTEB #6, 8k, 1024-d) and a Small-Cheap model (MiniLM, MTEB #40, 512-token context, 384-d) — are evaluated on a use case the user picks: general web Q&A, legal contracts, or long technical PDFs. Each carries its MTEB rank as a badge; clicking 'Run eval on my data' animates a recall@5 bar per model on ~30 of the user's own (query → doc) pairs and re-ranks them. The reveal: on general web the MTEB order holds, but on legal contracts the ranking reorders and the MTEB #1 drops to #2 (domain fit), and on long PDFs the small 512-token model collapses because it silently truncates 2k-token chunks (context-length mismatch). The lesson made tangible: MTEB is a shortlisting tool, not the decision — evaluate on your own data with recall@k, and reach for the cheaper levers before fine-tuning. Numbers are an illustrative simulation.

Mental-Model Corrections

"Pick whatever's #1 on MTEB." No — MTEB is a shortlisting tool; it's general-web and short-passage. Your data decides.
"Serious RAG means fine-tuning the embedder." Usually the opposite — chunking, hybrid, and reranking beat a custom model for most teams. Fine-tune last.
"Positive pairs are enough to fine-tune." They give marginal gains; hard negatives are where the real improvement comes from.
"Fine-tuning trains a model from scratch." No — you start from a strong pretrained model and nudge it (contrastive, on your pairs).
"I can swap embedding models anytime." Only by re-embedding your entire corpus — vectors aren't comparable across models. Choosing has a real switching cost.

Key Takeaways

The embedding model is the lens your whole knowledge base is retrieved through — choose deliberately (switching = re-embed everything).
Shortlist on MTEB by your real criteria (quality, dimensions, context length vs your chunks, multilingual, domain, license, cost) — then read it critically (general-web, short-passage).
Evaluate the shortlist on YOUR data (recall@k on ~30 real query→doc pairs). The model that finds your docs wins — this is the decisive step.
Fine-tune rarely — only after chunking/hybrid/reranking/prefixes, and only with a domain-vocabulary need + labeled pairs.
How: contrastive learning (MultipleNegativesRankingLoss, in-batch negatives) from a pretrained model, with hard negatives as the biggest lever.