Reranking & Late Interaction (Cross-Encoders, ColBERT)

Introduction

Both of the last two lessons ended pointing here. Hybrid search casts a wide, cheap net; RRF fuses the catch. But a wide net is noisy — the truly best document is often sitting at rank 7, not rank 1. Reranking is the precision stage that fixes that, and it's consistently the single biggest quality jump you can add to a RAG pipeline after hybrid search.

The core idea is a two-stage pipeline: a fast retriever optimizes for recall (don't miss anything), then a slow, far more accurate reranker optimizes for precision (put the best on top). Understanding why the reranker is more accurate — and what it fundamentally can't fix — is the heart of this lesson.

You'll learn:

The retrieve-then-rerank architecture and why two stages
Bi-encoder vs cross-encoder — the deep reason rerankers are more accurate
ColBERT / late interaction — the clever middle ground
The iron law: reranking improves precision, not recall (the recall ceiling)

Two Stages: Recall First, Precision Second

You cannot run an accurate-but-slow model over millions of documents per query — it would take forever. But you can run it over ~100. That single economic fact gives rise to the architecture behind virtually every serious search and RAG system today:

① Retrieve (fast · recall). A cheap first-stage retriever — bi-encoder (dense), BM25, or hybrid — scans the whole corpus and returns the top-N candidates (typically 50–100). Its job is recall: make sure the right documents are somewhere in that set, even if the ordering is rough.

② Rerank (slow · precision). A more expensive, more accurate model scores each of those N candidates against the query and reorders them, returning the top-k (typically 3–10) to the LLM. Its job is precision: get the genuinely best documents to the very top.

The division of labor is the point: stage 1 maximizes recall cheaply over millions; stage 2 maximizes precision expensively over ~100. As the saying goes, the cost of a bad retrieval far exceeds the cost of 50ms of reranking. Now — why is stage 2 so much more accurate?

An infographic titled 'Reranking & Late Interaction — Precision After Recall'. A pipeline flows left to right: a corpus of millions of documents, then stage one RETRIEVE (fast, recall) using a bi-encoder, BM25, or hybrid to cast a wide net cheaply and return the top-N candidates (about 50 to 100), then stage two RERANK (slow, precision) where a cross-encoder reads each query-document pair and reorders them to the top-k (about 3 to 10), then the LLM receives the top-k. A note explains you can't run the expensive model on millions of docs but you can on about 100 candidates — that's the whole point of two stages. Below, three architectures compared by how deeply query and document interact. Bi-encoder (stage 1, retrieval): encode query and doc separately into two vectors compared by cosine; they never interact because all meaning is crushed into one vector before comparison; wins on scalability (docs precomputed, ANN search over millions in milliseconds) but is least accurate (generic, averaged meaning). Cross-encoder (stage 2, the reranker): feed query and doc in together through one transformer with full cross-attention so every query token sees every doc token, producing one relevance score; most accurate and query-specific with little info loss, but slow — a full model pass per pair, no precompute. ColBERT (late interaction, the middle ground): one vector per token, scored by MaxSim where each query token finds its best-matching doc token and the maxima are summed, with doc tokens precomputed; gets token-level interaction AND scalability, but storage blows up, mitigated by ColBERTv2 quantizing about 6 to 10 times. A caveat: reranking improves precision, not recall — it only reorders what stage one retrieved, so no reranker can recover a doc the first stage missed (the recall ceiling); tune first-stage N first; rerankers include Cohere, Voyage, Jina, and open BGE at about 100 to 200 ms, and you should put the priciest model on the smallest candidate set. Banner: cast a wide net cheaply (recall), then sort it carefully (precision); reranking is the biggest quality jump after hybrid, but it can't find what retrieval missed.

Bi-Encoder vs Cross-Encoder: Why Rerankers Win

This is the conceptual core. The retriever and the reranker use fundamentally different architectures, and the difference explains everything.

🔵 Bi-encoder (what dense retrieval / embeddings use): the query and each document are encoded separately, each into its own fixed vector, and compared by cosine similarity. Crucially, the query and document tokens never interact — the model must compress all of a document's meaning into one vector before it has ever seen the query. It's fast and scalable (documents are embedded offline, so query time is just an ANN lookup), but that compression loses information and produces a generic, averaged representation.

🟢 Cross-encoder (every reranker is one): the query and document are concatenated into a single input — [query; doc] — and fed through one transformer, so cross-attention lets every query token attend to every document token. The model reads them together and outputs a single relevance score specific to this query. Far less information loss, far more nuance.

A useful mental image: a bi-encoder is two people who read the query and the document in separate rooms and then compare one-sentence summaries. A cross-encoder is one person reading both together, word against word. The second is obviously more accurate — and obviously slower.

from sentence_transformers import CrossEncoder

# Stage 1 already gave us ~50–100 candidates (dense / BM25 / hybrid). Now rerank them.
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")        # or Cohere/Voyage/Jina rerank APIs

query = "how do I dispute a duplicate charge?"
pairs = [(query, doc.text) for doc in candidates]          # EVERY query–doc pair…
scores = reranker.predict(pairs)                           # …a full transformer pass each (the cost)

reranked = [doc for _, doc in sorted(zip(scores, candidates), key=lambda x: -x[0])]
top_k = reranked[:5]                                       # hand only these to the LLM

# A bi-encoder embeds query & doc SEPARATELY (fast, precomputed).
# A cross-encoder reads them TOGETHER → far more accurate, but O(N) model calls → 2nd stage only.

And there's the catch in the code: the cross-encoder needs a full model pass for every query–document pair. You can't precompute anything (the score depends on the query), so it's O(N) transformer runs per query — fine for 100 candidates, impossible for 100 million. That's exactly why it lives in stage 2, never stage 1.

ColBERT & Late Interaction — the Middle Ground

Bi-encoders are fast but shallow; cross-encoders are accurate but unscalable. ColBERT stakes out a clever middle ground called late interaction.

Instead of pooling a document into one vector (bi-encoder) or re-running the model per pair (cross-encoder), ColBERT keeps a vector per token and scores with the MaxSim operator: for each query token, find its single best-matching document token (max cosine similarity), then sum those maxima across the query.

# ColBERT — "late interaction". Keep a vector PER TOKEN (not one pooled vector),
# then score with MaxSim: for each QUERY token, take its best match among DOC tokens, and sum.
import torch

def maxsim(query_tok_embs, doc_tok_embs):          # both: [num_tokens, dim], unit-normalized
    sim = query_tok_embs @ doc_tok_embs.T          # every query token × every doc token
    return sim.max(dim=1).values.sum().item()      # max over doc tokens, summed over query tokens

# Doc token embeddings are computed OFFLINE (like a bi-encoder → scalable),
# but scoring keeps token-level interaction (like a cross-encoder → accurate).
# Cost: one vector per token → huge index; ColBERTv2 quantizes it ~6–10×, PLAID speeds search.

Why it's clever: the document token vectors are computed offline — just like a bi-encoder, so it's scalable and ANN-friendly — yet scoring preserves token-level interaction between query and document, recovering much of the cross-encoder's nuance. "Late" interaction = the interaction is deferred to scoring time, after the (precomputed) encoding.

The price is storage: a vector for every token of every document is enormous. ColBERTv2 tames this with residual quantization (a centroid + a few low-bit residual bits), shrinking token embeddings from ~256 bytes to ~20–36 bytes (6–10× smaller), and PLAID makes search fast via centroid pruning. ColBERT can serve as a more accurate first-stage retriever or a reranker — and the late-interaction idea now extends to images (ColPali). Think of it as: bi-encoder scalability + (most of) cross-encoder accuracy, paid for in storage.

See It: The Recall Ceiling

Now the most important practical lesson — and it's best felt. Below, a fast first stage ranks 8 documents; the top-N pass to a cross-encoder reranker, which reorders them by true relevance. Drag N and watch what happens to the genuinely-relevant answers (⭐):

Drag N (candidates passed to the reranker). Watch the cross-encoder promote a buried-but-relevant doc (D6, stage-1 rank 6) to #1 — but only if N lets it through. Too small an N and it's lost: the recall ceiling.

Two things to take away from playing with it:

Reranking is powerful: the real answer D6 sits at first-stage rank 6 (its keywords didn't match well), but the cross-encoder — reading the full pair — recognizes its relevance and promotes it to #1. That's the precision win.
But it's helpless below the cutoff: set N too small and D6 (and D8) are cut before the reranker ever sees them. No amount of reranking genius can recover them. This is the recall ceiling, and it's the iron law of the next section.

The Iron Law: Precision, Not Recall

Burn this in, because it's the mistake that quietly caps so many RAG systems:

Reranking improves precision, not recall. A reranker can only reorder the candidates the first stage handed it — it can never recover a relevant document that retrieval missed.

Your pipeline's quality is therefore bounded by the first-stage recall ceiling. If the right document isn't in the top-N candidates, the best reranker on earth produces a beautifully-ordered list of wrong answers. The practical consequences:

Fix recall first. If reranked results are poor, don't reach for a fancier reranker — check whether the right docs are even in the candidate set. Improve the first stage (hybrid search, better embeddings, bigger N) until recall is high.
Choose N deliberately. Larger N → higher recall ceiling but more rerank latency/cost; smaller N → cheaper but risks cutting good docs. Common range: top 50–100 into the reranker.
Layer cost by candidate count — the golden rule of multi-stage retrieval: put the most expensive technique on the fewest candidates. Cheap recall (BM25/dense/hybrid) over millions → cross-encoder over ~100 → (optionally) an even pricier LLM reranker over the top handful, only if the use case demands it.

Choosing a Reranker (2026)

You almost never train a reranker — you pick one. The landscape:

Managed cross-encoder APIs: Cohere Rerank (v3.5 / v4, broad language coverage), Voyage rerank-2.5, Jina Reranker v3 (notably low latency, ~188ms). Zero ops, you pay per query.
Open-source (self-host): BGE reranker v2-m3 (Apache-2.0, strong and free to run), mixedbread mxbai-rerank, Sentence-Transformers cross-encoders. Full control, your hardware.
LLM rerankers: RankGPT / RankZephyr and listwise methods prompt an LLM to order the candidates. Often the most accurate and the most expensive — reserve them for the top handful of candidates.

Rough economics to anchor on: a cross-encoder rerank of ~30 candidates adds ~100–200ms; open models like BGE land around ~145ms at a fraction of a cent per thousand queries, managed APIs a bit slower and pricier but often higher quality. As always — benchmark a couple on your own data (nDCG / recall@k, and latency budget) rather than trusting a leaderboard. The category (add a reranker) matters far more than the exact model.

🧪 Try It Yourself

Drive the funnel, then reason like an engineer:

In the widget, start at the default N and increase it slowly. At what N does ⭐D6 first reach the final top-k? What about ⭐D8? What's happening below those values?
Why can the reranker rank D6 #1 when the first stage buried it at rank 6? What does the cross-encoder see that the bi-encoder didn't?
Your reranked answers are still wrong. A teammate wants to try a more expensive reranker. Based on the iron law, what should you check first?
Why is a cross-encoder a terrible choice for stage 1 (searching the whole corpus), but ideal for stage 2?
You need cross-encoder-level accuracy but also a scalable index you can search directly. Which architecture fits, and what's its main cost?

→ (1) D6 enters the top-k once N ≥ its stage-1 rank (6); D8 needs N ≥ 8; below those, they're cut before reranking (recall ceiling) and no reorder can recover them. (2) The cross-encoder reads the query and D6 together with full cross-attention, so it judges relevance to this specific query — the bi-encoder had compressed D6 into a generic vector with no view of the query. (3) Recall, not the reranker — verify the right docs are even in the candidate set; if not, fix the first stage / raise N. A better reranker can't reorder what isn't there. (4) Stage 1 needs O(1)-per-doc, precomputable scoring over millions; a cross-encoder is O(N) full-model passes with no precompute — fine for ~100 candidates, impossible for millions. (5) ColBERT (late interaction) — token-level accuracy with precomputed, scalable doc embeddings; main cost = storage (one vector per token; mitigated by ColBERTv2 quantization).

Mental-Model Corrections

"Reranking improves recall." No — it improves precision (ordering). It can only reorder what stage 1 retrieved; the first-stage recall ceiling is the hard limit.
"Embeddings and rerankers are the same kind of model." Different architectures: embeddings are bi-encoders (query & doc encoded separately); rerankers are cross-encoders (encoded together, full cross-attention).
"Just use the accurate model for everything." A cross-encoder is O(N) full passes per query with no precompute — impossible over millions of docs. It's a second stage, by necessity.
"ColBERT is just another embedding model." It's multi-vector (one per token) with MaxSim late interaction — more accurate than a bi-encoder, scalable unlike a cross-encoder, but storage-hungry.
"A better reranker fixes bad results." Often the fix is better first-stage recall (hybrid, bigger N), not a fancier reranker.
"Rerank everything you retrieved." Put the priciest model on the fewest candidates — recall cheaply over millions, rerank ~100, LLM-rerank only the top few.

Key Takeaways

Retrieve-then-rerank is the standard two-stage pattern: stage 1 (fast, recall) returns top-N (~50–100) via bi-encoder/BM25/hybrid; stage 2 (slow, precision) reorders to top-k (~3–10) with a reranker.
Bi-encoder encodes query & doc separately (fast, scalable, less accurate); cross-encoder encodes them together with full cross-attention (most accurate, but O(N) per query → second stage only). Every reranker is a cross-encoder.
ColBERT / late interaction = a vector per token + MaxSim, with doc tokens precomputed — cross-encoder-ish accuracy and bi-encoder scalability, paid for in storage (ColBERTv2 quantizes 6–10×).
The iron law: reranking improves precision, not recall — it can't recover what the first stage missed. Fix recall first, choose N deliberately, and put the most expensive model on the fewest candidates.
Pick a reranker (Cohere/Voyage/Jina managed, BGE open, LLM rerankers for the top few) and benchmark on your data — the category matters more than the model.
Next: query rewriting & HyDE — instead of fixing results after retrieval, we improve the query itself before it ever hits the index.