Choosing the Right Vector Store

You've met the index algorithms (HNSW/IVF), toured the database landscape, and learned filtering and ops. Now the decision every RAG project faces: which vector store do I actually use?

This lesson is a decision framework — not "the best database" (there isn't one), but a repeatable process to pick the right one for your scale, stack, features, and team, and to avoid the two classic mistakes: over-engineering (a distributed cluster for 50K docs) and painful migrations (outgrowing a choice and having to re-embed everything).

And we start with the biggest 2026 shift, because it changes the whole conversation: vectors are now a data type, not a database category. That single fact reframes the very first question you should ask.

In this lesson you'll learn:

• The first question: do you even need a dedicated vector DB?
• The axes that actually decide it
• A concrete choice ladder by scale
• How to test before committing — and the migration cost to respect

For years, "add semantic search" meant "stand up a vector database." In 2026 that's often the wrong first move, because the major databases you already run now have native vector search built in:

• PostgreSQL → pgvector (+ pgvectorscale) — now comfortable to ~50M vectors.
• MongoDB → Atlas Vector Search. Oracle, Elasticsearch, Redis → native vector types too.

So vectors went from being a database you bolt on to being a column in the database you have. The practical consequence:

If you already run Postgres/Mongo/Elastic and you're under tens of millions of vectors, you very likely don't need a separate vector DB. Keep vectors next to your app data — fewer systems, transactional consistency, one less thing to operate.

Dedicated vector databases now earn their place mainly at large scale, strict sub-50ms latency, or heavy filtering/hybrid needs. Default bias: use what you already have, and add a dedicated store only when you've outgrown it.

If you do need to choose, these are the questions that matter — roughly in order of weight:

• Scale (number of vectors). The single biggest axis — it sets the shortlist (below).
• Existing stack / data-platform commitment. Already standardized on Postgres or Mongo? That's a heavy thumb on the scale toward pgvector / Atlas — pick by your platform first.
• Managed vs self-hosted. Your ops capacity and cost: managed (Pinecone) = zero ops, you pay; self-hosted (Qdrant/Weaviate/Milvus) = you run it, cheaper at scale (the cost advantage of self-hosting becomes decisive above ~50M vectors).
• Filtering / multi-tenancy. This is where databases diverge most — if you filter heavily, it's a top criterion (and you must test it).
• Hybrid search. Need keyword + vector in one query? → Weaviate has it built-in.
• Latency / QPS. Sub-50ms p99 at high QPS narrows the field (Pinecone, Milvus, tuned Qdrant).
• Compliance / deployment. On-prem or data-residency requirements push you to self-hosted open-source.

Concrete, current guidance — find your scale, then apply the modifiers:

< ~10M vectors (most apps). pgvector if you're on Postgres; Chroma for local/prototyping. (And remember — at a few thousand vectors, plain numpy/flat is genuinely fine.) Don't over-engineer.

10M–50M. pgvector + pgvectorscale still works well; reach for Qdrant if you need heavy filtering / multi-tenancy — and test your real filter combinations before committing.

50M–500M. Qdrant (great DX + filtering) or Milvus (best operational scaling); Pinecone if you want fully managed with sub-50ms latency. (Self-hosting's cost advantage is decisive in this range.)

500M+ / billions. Milvus / Zilliz Cloud (distributed — the complexity tax earns its keep) or Pinecone (managed serverless).

Modifiers (at any scale): need hybrid search → Weaviate; want zero ops and will pay → Pinecone; on-prem/compliance → self-hosted OSS; heavy filtering → Qdrant.

As with model selection, turn the debate into a scorecard — candidates as rows, your axes as columns — so the decision is explicit and revisitable:

Weight the columns by your priorities, score 2–3 finalists, and write down why you chose — so when you revisit in a year (or hit a new constraint), the reasoning is there. A clear scorecard also makes it obvious when "pgvector, because we already run Postgres and we're at 3M vectors" is the right, boring, correct answer.

Two final, crucial habits:

1. Benchmark on your data — don't choose from a blog post. Vendor numbers are run on their favorable datasets. Load a slice of your corpus and measure what you care about: recall@k, latency at your QPS, and — most importantly — your actual filter combinations (filtering is where databases diverge most, and where a great-on-paper DB can fall apart on your workload). An afternoon of benchmarking beats months of regret.

2. Respect the switching cost. Migrating vector stores isn't a config change — it usually means re-embedding and re-indexing your entire corpus (vectors aren't portable across DBs/models, and index formats differ). So choose with some headroom for growth — but don't over-provision for a scale you may never reach (a billion-vector architecture is wasted effort on a 2M-vector app). The balance: start where you are, design so migration is possible, and move only when the numbers force it.

Run the decision on each project — name the store and the reason:

1. A B2B SaaS already on Postgres, ~4M document chunks, needs per-tenant filtering. → ?
2. A startup with 80M product vectors, a small team that wants zero ops, needs sub-50ms. → ?
3. A weekend prototype: chat with 300 PDFs locally. → ?
4. An on-prem enterprise deployment, ~120M vectors, heavy metadata filtering, must self-host. → ?

→ 1: pgvector (already in your Postgres, <50M, transactional — don't add a system; WHERE tenant_id does the filtering). 2: Pinecone (managed, sub-50ms, scales — worth paying for zero ops). 3: Chroma (local, instant). 4: Qdrant (self-host OSS, strongest filtering at that scale). Notice the answer is driven by stack + scale + filtering + ops — never "the highest-ranked DB."

• "There's a best vector database." No — only the best for your scale, stack, filters, latency, and team. It's a fit decision, not a ranking.
• "I need a dedicated vector DB." In 2026, vectors are a data type — if you run Postgres/Mongo and you're under ~50M vectors, use its native search. Add a dedicated store only when you outgrow it.
• "Pick by the vendor benchmark." Vendor numbers use favorable data — test on your own corpus and filters (where DBs diverge most).
• "Over-provision so I never migrate." Don't build for a billion vectors at 2M — but design so migration is possible, since switching = re-embed + re-index everything.
• "Managed vs self-hosted is about quality." It's ops vs cost/control — both can be excellent; self-host's cost edge shows above ~50M.

• First question: do you even need a dedicated vector DB? In 2026 vectors are a data type — on Postgres/Mongo/Elastic under ~50M vectors, use native vector search (pgvector/Atlas). Don't add a system.
• If you do choose, decide on the axes: scale, existing stack, managed-vs-self-hosted (ops/cost), filtering/multi-tenancy, hybrid, latency, compliance.
• Ladder by scale: <10M pgvector/Chroma · 10–50M pgvector+scale/Qdrant · 50–500M Qdrant/Milvus/Pinecone · 500M+ Milvus/Zilliz/Pinecone. Modifiers: hybrid→Weaviate, zero-ops→Pinecone, filtering→Qdrant.
• Make it a scorecard, test on your data (especially your filter combinations), and respect the migration cost (re-embed + re-index).
• Start simple; migrate only when the numbers force it — and that closes the Vector Databases section. Next: building the full RAG pipeline.