Similarity Metrics (Embedding, BLEU/ROUGE — and Their Limits)

Introduction

The last lesson, Functional Correctness & Exact Match, hit a wall: strict == can't see that "thirty days" means 30 days, or that two differently-worded summaries are both fine. When the output isn't verifiable and exact match is too rigid, you move up the ladder to similarity metrics — graders that score how close an output is to a reference, on a continuous scale instead of pass/fail.

There are two families. Lexical metrics (BLEU, ROUGE) count overlapping words. Semantic metrics (embeddings, BERTScore) compare meaning in vector space. Both are genuinely useful — and both come with a sharp limit you must understand before you trust a single one of their numbers. This lesson is the family, how each works, and exactly where each one lies to you.

An infographic titled 'Similarity Metrics: BLEU, ROUGE, Embeddings — and Their Limits' showing the family of fuzzy graders you reach for when exact match is too strict, arranged as a ladder of increasing sophistication and cost. The first rung is LEXICAL OVERLAP: BLEU, which is precision-based and counts what fraction of the candidate's n-grams appear in the reference with a brevity penalty for being too short, used for machine translation; and ROUGE, which is recall-based and counts what fraction of the reference's n-grams the candidate covered, used for summarization; both are fast and deterministic but blind to meaning, so a correct paraphrase like returns accepted within thirty days scores low against you have thirty days to return it, a false fail. The second rung is SEMANTIC SIMILARITY: embedding-based metrics and BERTScore embed both texts and compare them with cosine similarity, capturing paraphrases and synonyms that n-grams miss and correlating better with human judgment. But a central warning panel states the deeper trap that catches both families: SIMILAR IS NOT CORRECT. A negation flip, the medication is not safe for children versus the medication is safe for children, scores high on both n-gram overlap and embedding cosine even though the meaning is opposite, the antonymy problem where best and worst are neighbors in embedding space; a wrong number, three hundred dollars versus thirty dollars, also scores high; and a fluent hallucination, the Eiffel Tower was completed in 1925 versus 1889, scores high because embeddings cannot tell a confident falsehood from the truth. Two more gotchas are noted: these metrics are reference-based, assuming a single correct answer that is unrealistic for open-ended tasks; and cosine scores are hard to threshold because embedding spaces are anisotropic and cluster in a narrow band. The guidance for when to use them is that they are fast, cheap, deterministic, and interpretable, good for machine translation, summarization coverage, and cheap regression signals, but only as one signal among many, never the sole gate. The metric ladder is shown progressing exact match, then n-gram BLEU and ROUGE, then embedding and learned metrics, then an LLM judge, each looser, smarter, and more expensive. The takeaway banner reads: similarity metrics measure how close an output is to a reference, not whether it is correct, so use them as cheap signals and escalate to a judge when you need truth.

Lexical Overlap: BLEU & ROUGE

The classic metrics measure n-gram overlap — how many word-sequences the candidate and reference share. They're a mirror image of each other:

BLEU (Bilingual Evaluation Understudy) is precision-oriented: of the n-grams I generated, what fraction appear in the reference? It adds a brevity penalty so you can't game it by emitting one safe word. It's the long-standing default for machine translation.
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is recall-oriented: of the n-grams in the reference, what fraction did I cover? That makes it the default for summarization, where coverage matters more than exact phrasing.

Under the hood, both are just counting:

from collections import Counter

def ngrams(text, n=1):
    toks = text.lower().split()
    return Counter(tuple(toks[i:i + n]) for i in range(len(toks) - n + 1))

def overlap(cand, ref, n=1):
    c, r = ngrams(cand, n), ngrams(ref, n)
    shared = sum((c & r).values())                      # clipped n-gram matches
    bleu_precision = shared / max(1, sum(c.values()))   # BLEU: of MY n-grams, how many are in the ref?
    rouge_recall   = shared / max(1, sum(r.values()))   # ROUGE: of the REF's n-grams, how many did I cover?
    return round(bleu_precision, 2), round(rouge_recall, 2)

overlap("returns accepted within 30 days", "you have 30 days to return it")
# -> (0.40, 0.29)   a CORRECT paraphrase, yet only "30" and "days" line up

That's the whole idea: BLEU asks “is what I said in the reference?”, ROUGE asks “did I say what's in the reference?” Both are fast, deterministic, cheap, and interpretable — and, as the result hints, both are about to disappoint you.

The Lexical Trap: Blind to Meaning

Notice what just happened: "returns accepted within 30 days" and "you have 30 days to return it" are the same answer, yet they share almost no n-grams, so BLEU/ROUGE score them low. That's a false fail — and it's the core weakness of lexical metrics: they measure surface form, not meaning. They're blind to synonyms, paraphrases, and word order, which is why they correlate only weakly with human judgment on open-ended text. (METEOR was invented partly to patch this, adding stemming and synonym matching for better correlation — but it's still fundamentally lexical.)

Worse, the failure runs both ways. Because they only count word overlap, you can add a few negation words to a reference and produce a sentence with the opposite meaning that still scores a high BLEU — almost every n-gram is preserved. Lexical metrics can fail a correct answer and pass a wrong one. That's a shaky foundation to gate a release on.

Semantic Similarity: Embeddings & BERTScore

To get past surface form, compare meaning instead of words. Embed both texts into vectors (the same embedding machinery from the RAG container) and measure their cosine similarity — paraphrases land close together, so "returns within 30 days" ≈ "you have 30 days to return it" even with no shared words.

BERTScore does this at the token level: it embeds each token in context (so “bank” differs in “river bank” vs “bank account”), matches candidate tokens to reference tokens by cosine similarity, and aggregates into precision/recall/F1. BLEURT goes further — a learned metric, fine-tuned to predict human ratings. These semantic and learned metrics correlate notably better with human judgment than n-gram overlap, and they fix the paraphrase problem outright.

This feels like the answer. It mostly isn't — and the reason is the most important idea in this lesson.

See It: Does the Metric Agree with the Truth?

Put all three graders side by side. For each pair below, you'll see the exact-match, n-gram, and embedding scores — then reveal whether the candidate is actually a good answer, and watch which graders got fooled.

Interactive: pick a (candidate, reference) pair and compare exact match, n-gram overlap (BLEU/ROUGE), and embedding cosine — then reveal the ground truth. A correct paraphrase makes n-gram false-fail while embeddings catch it; a negation flip, a wrong number, and a fluent hallucination all make BOTH n-gram and embeddings false-pass. The throughline: similar is not correct.

The pattern is unmistakable. Embeddings rescue the paraphrase that n-grams failed — a real upgrade. But the negation, the wrong number, and the fluent hallucination sail past both with high scores. Every grader here answers “how similar?” — and none of them answers “is it right?”

The Deeper Trap: Similar ≠ Correct

Here is the limit that catches every metric in this lesson: similarity is not correctness. A high BLEU, ROUGE, or BERTScore tells you an output resembles the reference — not that it's true. Two failure modes make this concrete:

The antonymy problem. In embedding space, “safe” and “not safe” (or “best” and “worst”) are neighbours — they appear in the same contexts. So a negation flip, which inverts the meaning, barely moves the cosine score. The embedding shrugs at the one word that matters.
Fluent falsehoods. A confident, well-formed wrong answer (“completed in 1925” for a fact that's 1889) is semantically coherent and lexically close — so it scores high. The metric "doesn't distinguish confident falsehoods from accurate information."

Watch all four rungs of the ladder grade the same negation — and only the last one catch it:

from anthropic import Anthropic
client = Anthropic()

# The grader LADDER for one (candidate, reference) pair -- each rung looser & smarter:
cand = "The medication is not safe for children."
ref  = "The medication is safe for children."

exact = cand.strip().lower() == ref.strip().lower()    # -> False
bleu  = overlap(cand, ref)[0]                          # -> 0.83   almost all words shared!
cos   = cosine(embed(cand), embed(ref))                # -> 0.94   antonyms are neighbours!
#   ...all three say "very similar" -- but the candidate means the OPPOSITE.

# For CORRECTNESS (not similarity), escalate to a judge -- Section 3:
verdict = client.messages.create(
    model="claude-opus-4-8", max_tokens=10,
    system="Reply CORRECT or WRONG: does the candidate match the MEANING of the reference?",
    messages=[{"role": "user", "content": f"Reference: {ref}\nCandidate: {cand}"}]
).content[0].text                                      # -> "WRONG"  it caught the negation

Exact match, BLEU, and embedding cosine all call the negated sentence “very similar.” Only the LLM judge, asked about meaning and correctness rather than overlap, returns WRONG. That's the boundary of this lesson — and the whole reason LLM-as-a-Judge (the next section) exists.

Two More Gotchas: Reference-Dependence & Anisotropy

Even setting truth aside, two practical traps bite:

They're reference-based. Every metric here needs a gold answer to compare against, and implicitly assumes there's one right answer. For open-ended tasks with many valid responses (the lesson of Why Evaluating LLMs Is Uniquely Hard), that assumption breaks — your output can be excellent and simply unlike your single reference.
Cosine is hard to threshold. Embedding spaces are anisotropic — vectors cluster in a narrow cone, so cosine scores bunch up (often everything lands around 0.7–0.9). Is 0.82 a pass? There's no clean answer, the band shifts with every embedding model, and even unrelated sentences can score deceptively high. A raw cosine number is far mushier than it looks.

These don't make the metrics useless — but they're why a similarity score is a signal, never a verdict.

So When DO You Use Them?

Don't throw them out — place them correctly. Similarity metrics are fast, free, deterministic, and interpretable, which earns them real jobs:

Machine translation → BLEU, where good references exist and adequacy is largely lexical.
Summarization coverage → ROUGE, to check you included the key content.
Cheap regression signals → a quick "did this output drift far from the known-good reference?" check in CI, run on every change.
One signal among many → combined with deterministic checks and (where needed) a judge — never the sole gate.

Think of it as a ladder, the one running through this whole section: exact match → n-gram (BLEU/ROUGE) → embedding / learned (BERTScore/BLEURT) → LLM judge. Each rung is looser, smarter, more expensive, and less interpretable than the last. Start at the bottom; climb only as far as the task forces you. Most teams should reach for similarity metrics as a cheap early signal — and reach for a judge the moment correctness, not resemblance, is what they actually need to measure.

🧪 Try It Yourself

Make the limits undeniable with a two-minute experiment on a pair from your own system:

Take a (candidate, reference) pair and compute a rough word-overlap (even by hand: shared words ÷ total) and, if you can, an embedding cosine. Note both numbers.
Make a correct paraphrase of the candidate — reword it completely without changing the meaning. Recompute. Watch the n-gram score crash while the meaning is unchanged. (False fail.)
Make a negation or wrong-number flip — insert a not, or change one number — so the answer is now wrong. Recompute. Watch both scores stay high. (False pass.)
Ask yourself: could either number, alone, have caught step 3? If not, you've just proven why similarity ≠ correctness — and why you'll need a judge for the things that actually matter.

Keep the pair; it's a perfect demo for the next time someone proposes gating a release on BLEU.

Mental-Model Corrections

“A high BLEU/ROUGE means a good output.” No — they measure lexical overlap, not quality. A correct paraphrase scores low; a negated sentence scores high. Weak correlation with human judgment on open-ended text.
“Embeddings fix the problem.” They fix the paraphrase problem (a real upgrade), but inherit the antonymy problem — negations and antonyms stay close in vector space — and happily pass fluent falsehoods.
“High similarity = correct.” The central trap of the whole lesson. Similarity is resemblance to a reference; correctness is truth. No metric here measures truth.
“These are objective ground-truth metrics.” They're reference-based — they need a gold answer and assume there's only one. For open-ended tasks, that assumption fails.
“A cosine of 0.82 clearly means it's good.” Cosine is hard to threshold — anisotropic spaces make scores cluster, and the band shifts per model. Treat it as a fuzzy signal, not a verdict.
“BLEU and ROUGE are obsolete.” No — they're still the right, cheap tool for MT, summarization coverage, and fast regression signals. Just never the sole gate, and never for correctness.

Key Takeaways

When exact match is too strict, climb to similarity metrics — continuous scores of how close an output is to a reference.
Lexical (BLEU/ROUGE) count n-gram overlap: BLEU = precision (MT), ROUGE = recall (summarization). Fast and deterministic, but blind to meaning — they fail correct paraphrases and can pass negated nonsense.
Semantic (embeddings, BERTScore, BLEURT) compare meaning via cosine of contextual embeddings — they fix paraphrases and correlate better with humans, but inherit the antonymy problem and pass fluent falsehoods.
The one idea to keep: similar ≠ correct. Every metric here measures resemblance, not truth; none catches a negation, a wrong number, or a confident hallucination reliably.
They're also reference-based (assume one right answer) and cosine is hard to threshold (anisotropic clustering). Treat their output as a signal, not a verdict.
Use the ladder: exact → n-gram → embedding/learned → LLM judge, each looser, smarter, and costlier. Similarity metrics are great cheap signals for MT, summarization, and regression — but the moment you need correctness, escalate to a judge (next section).