Building a Semantic Search Engine from Scratch

Introduction

You've learned what embeddings are (L70), how to compare them (L71), and how to choose a model (L72). Now you build the thing all of that was for: a semantic search engine — search by meaning, not keywords — in about 20 lines of plain Python. No framework. No vector database. No magic.

This is the payoff lesson, and it's worth doing by hand because every RAG system, every vector database, every "chat with your docs" product is this exact loop underneath. Build the core yourself and the rest of the container becomes "how do we make this fast, accurate, and production-ready?" — not "what is this black box?"

In this lesson you'll build:

The 3-step engine: index → query → rank
A working search() in ~20 lines of numpy
Proof that it matches by meaning (queries that share zero words with the answer)
An honest "no good match" threshold — and why this won't scale (→ vector databases next)

The Whole Idea in Three Steps

Semantic search is shorter than people expect. It's three steps — and the first one happens once, offline:

Index (once): embed every document with your model, normalize the vectors, and store them as a matrix. This is your search index.
Query: when a user searches, embed their query — with the same model — into one vector.
Rank: compute the similarity (cosine) between the query vector and every document vector, and return the top-k highest-scoring docs.

That's the entire algorithm. Because we normalize (last lesson), cosine similarity is just a dot product — so "compare the query to all N documents" is a single matrix-vector multiply. Here's the shape of it:

An infographic titled 'A Semantic Search Engine in 3 Steps'. Step 1 INDEX (once, offline): a list of docs is embedded and normalized into doc_vecs, an N by d matrix. Step 2 QUERY (same model): the query 'how do I get my money back?' is embedded and normalized into a 1 by d vector q. Step 3 RANK (one matmul): scores = doc_vecs dot q (cosine), shown as ranked bars — Refunds 0.62 (green, top), Free shipping 0.18, Office hours 0.09 — then return top-k. A green callout: the query 'get my money back' and the answer 'Refunds are processed...' share zero words, yet cosine 0.62 ranks it number 1 — that's meaning, not keywords. A red warning: this is brute-force, every query compares against every vector, O(n) per query, great to about a few thousand docs; beyond that you need approximate nearest neighbor (ANN) plus a vector database. A bottom banner: embed once, embed the query, rank by cosine, return the top matches — that's the core of every RAG system.

Step 1 — Index the Corpus

First, an embed() helper that returns normalized vectors (so cosine = dot), then embed the whole corpus once. We'll use OpenAI here (its embeddings don't need role prefixes; for an open model like E5/BGE you'd add query: / passage: from L70).

import numpy as np
from openai import OpenAI
client = OpenAI()

def embed(texts):
    r = client.embeddings.create(model="text-embedding-3-small", input=texts)
    v = np.array([d.embedding for d in r.data])
    return v / np.linalg.norm(v, axis=1, keepdims=True)   # normalize -> cosine == dot

# --- the 'knowledge base' (your docs) ---
docs = [
    "Refunds are processed within 5 business days to your original payment method.",
    "To reset your password, click 'Forgot password' on the login screen.",
    "Our office is open Monday to Friday, 9am to 6pm.",
    "Free shipping applies to all orders over $50.",
]

# --- STEP 1: INDEX (do this ONCE, offline) ---
doc_vecs = embed(docs)        # shape (4, 1536), each row a normalized vector
print(doc_vecs.shape)         # (4, 1536)

Steps 2 + 3 — Search (Embed → Rank → Top-k)

Now the search itself. Embed the query the same way, take the dot product against every doc (that's cosine, since everything's normalized), and return the highest scores. The @ is one matrix-vector multiply — all comparisons at once.

# --- STEPS 2 + 3: SEARCH ---
def search(query, k=2, threshold=0.30):
    q = embed([query])[0]                 # STEP 2: embed the query (same model!)
    scores = doc_vecs @ q                 # STEP 3: cosine vs every doc — one matmul
    top = np.argsort(-scores)[:k]         # indices of the top-k highest scores
    return [(docs[i], float(scores[i])) for i in top if scores[i] >= threshold]

for doc, score in search("how do I get my money back?"):
    print(f"{score:.2f}  {doc}")

# 0.62  Refunds are processed within 5 business days...   <- found it

That's a working semantic search engine. ~20 lines, no framework. argsort(-scores) sorts descending; [:k] keeps the top k. Everything a vector database does at its core, you just wrote.

See It Work: Meaning Beats Keywords

Look closely at what just happened. The query was "how do I get my money back?" — and the winning document was "Refunds are processed within 5 business days." They share no words at all — not "money," not "back," not "refund." A classic keyword search would return nothing.

Semantic search found it because the two sentences point in nearly the same direction in embedding space (cosine 0.62) — they mean the same thing. Try a few more and watch it generalize:

"when are you open?" → the office-hours doc
"I forgot my login" → the password-reset doc
"do you deliver for free?" → the shipping doc

None of those share the key noun with their answer. That is why semantic search beats keyword search for real questions — and why it's the retrieval backbone of RAG.

Making It Honest: the "No Good Match" Threshold

A subtle but critical detail for anything feeding an LLM: a naive search always returns its top-k — even when nothing in your corpus is actually relevant. Ask our engine "what's the capital of France?" and it will still hand back its least-bad doc, with a low score.

In plain search that's a poor result. In RAG it's dangerous: you'd stuff an irrelevant document into the prompt and invite a confident, wrong answer. The fix is the threshold already in our search():

return [... for i in top if scores[i] >= threshold]   # drop weak matches

If even the top result is below the bar (say, cosine < 0.3), return nothing — and let the app say "I don't have information on that" rather than grounding on junk. (Tuning that threshold is a real eval task; for now, internalize the principle: a good retriever knows when to return nothing.)

Why This Won't Scale (and What's Next)

What you built is exact / brute-force search: every query computes a similarity against every vector in the corpus — that's O(n) per query. With 4 docs it's instant; with a few thousand it's still fine; but at hundreds of thousands or millions of vectors, scanning all of them on every query becomes too slow and memory-hungry for interactive use.

The fix is the entire next section: approximate nearest neighbor (ANN) indexes (HNSW, IVF) and the vector databases built around them. They trade a tiny bit of accuracy for enormous speedups — finding the near-top matches in milliseconds across millions of vectors, without comparing against every one.

And brute-force isn't the only thing real systems add. Coming up in this container: chunking (documents are too big to embed whole), metadata filtering, hybrid search (keyword + semantic), and reranking (a sharper second-pass scorer). You've built the core; the rest of RAG is making it fast, accurate, and robust.

🧪 Try It Yourself

Your turn — extend the engine. Take the ~20 lines above and:

Add your own docs (5–10 facts about your life or project) and search them with questions that share no keywords. Does it still find the right one?
Trigger the threshold: search for something not in your corpus (e.g. "what's the weather?"). Confirm it returns nothing — then lower the threshold to 0.0 and watch it return junk. That gap is why thresholds matter.
Swap the model (e.g. a local sentence-transformers model) and confirm you must re-embed the whole corpus (vectors aren't comparable across models — L72).

Twenty lines, and you've built the heart of every RAG system. Everything ahead just makes it production-grade.

Interactive: a working semantic-search playground. A fixed 6-document support corpus (refunds, office hours, password reset, free shipping, payment methods, HQ) is searched by picking one of five queries. The engine scores the query against every document by cosine and ranks them live, and a 'no good match' threshold slider lets the user drop weak matches. The two lessons are felt directly: the winning doc for 'how do I get my money back?' is 'Refunds are processed within 5 business days' at cosine 0.62 while sharing ZERO words with the query (shared-word count computed live) — meaning, not keywords; and searching 'what's the capital of France?' at threshold 0.30 correctly returns NOTHING (the app says it doesn't know), while dragging the bar to 0.00 makes it hand back its least-bad doc — junk that in RAG becomes a confident wrong answer. Every doc is scored on every query, illustrating the O(n) brute-force scan that ANN indexes fix next. Scores are an illustrative simulation.

Mental-Model Corrections

"Semantic search is keyword search with extra steps." No — it matches by meaning/direction, so it finds answers that share zero words with the query (and misses nothing just because the wording differs).
"I can embed the query with any model." It must be the same model (and same prefix convention) as the documents — otherwise the vectors aren't comparable.
"Top-k is always a good answer." Not without a threshold — a naive retriever returns its least-bad doc even when nothing is relevant. A good retriever can return nothing.
"This scales to millions of docs." This brute-force version is O(n)/query — fine to ~thousands; past that you need ANN + a vector DB (next section).
"I need a vector database to start." No — for a few thousand docs, numpy is genuinely enough. Add a vector DB when scale forces it, not before.

Key Takeaways

A semantic search engine is 3 steps: index (embed all docs once, normalized) → query (embed it, same model) → rank (cosine vs every doc, return top-k) — ~20 lines of numpy.
Because vectors are normalized, cosine = dot, so ranking all N docs is one matrix-vector multiply.
It matches by meaning: it finds answers that share no keywords with the query — the reason semantic search powers RAG.
Always add a similarity threshold so the retriever can return nothing when nothing is relevant (critical before feeding an LLM).
This is brute-force (O(n)/query) — great to ~thousands of docs; beyond that, ANN indexes + vector databases (the next section). You've built RAG's core.