Inserting, Updating & Scaling Your Index

Introduction

So far your vector index has been a static snapshot — embed once, search forever. Real data isn't like that: documents get added, edited, and deleted constantly, and your index has to keep up without going stale, leaking deleted data, or falling over as it grows.

This is the operational side of vector search — the part that separates a demo from a system you can run for years. Most courses stop at "search"; here we cover keeping the index fresh (insert/update/delete) and scaling it (memory, sharding, re-indexing). Get it wrong and you ship stale answers, fail a compliance audit (data that won't delete), or watch latency collapse under load.

In this lesson you'll learn:

The ingestion pipeline — batch upserts and deterministic IDs
Why updates mean re-embedding (and the stale-index trap)
Why deletes are tombstones — and the compliance angle
Re-indexing/compaction to stay healthy, and scaling (compress, shard, replicate)

The Ingestion Pipeline (Insert / Upsert)

Getting vectors in is a pipeline: source → chunk → embed → upsert (with metadata). Two practices make it production-grade:

Batch your embeddings. Embedding (and upserting) one item at a time is slow and expensive — APIs and indexes are far more efficient in batches of dozens-to-hundreds. Always batch.
Use deterministic IDs. Derive each vector's ID from its content's identity — e.g. hash(source + chunk_index). Then upsert (insert-or-update) means re-ingesting the same document overwrites its old vectors instead of creating duplicates. Random IDs guarantee a duplicate mess on every re-run.

("Upsert" = insert if new, update if the ID exists — the workhorse operation.)

An infographic titled 'Keep Your Index Fresh — and Scalable' with two panels. The indigo THE LIFECYCLE (data lives) panel has three operations: INSERT/UPSERT (add new vectors) — chunk, embed in batches, upsert with metadata, and use deterministic IDs so re-ingesting updates rather than duplicates; UPDATE (a doc changed) — re-embed the changed chunks and upsert by ID to overwrite, with a warning that editing content but not the vector makes the index silently lie (stale); DELETE (remove a doc or user data) — graph indexes can't cheaply drop a node so they tombstone (soft delete), hiding it from results now and reclaiming space by compaction later, and GDPR may need a hard delete plus rebuild. The amber SCALING (memory is the bottleneck — vectors and index live in RAM) panel lists four levers: Compress (quantize for more per machine), Shard (split across nodes, query all and merge), Replicate (copies for more QPS and failover), and Disk/DiskANN (huge corpora on cheaper hardware); plus a note to re-index periodically because HNSW graphs fragment as tombstones pile up and IVF centroids drift, using blue-green to serve a static copy while rebuilding; and a note that managed or serverless databases like Pinecone handle sharding, replication, and re-indexing for you. A bottom banner: search is the easy part — keeping the index fresh, correct, and affordable as data lives and grows is production RAG.

Updates: Re-embed, Don't Just Edit the Text

When a document changes, you can't just edit the stored text — the vector was computed from the old content. You must re-embed the changed chunks and upsert them by ID, so the new vector overwrites the old.

This is the stale-index trap (recall the embedding-pitfalls lesson): if you update a document's content but forget to re-embed its vector, the index silently lies — it keeps retrieving (and ranking) that doc based on what it used to say. No error, just wrong results.

Do it incrementally, not by rebuilding everything: track a content hash or updated_at per chunk, and re-embed only what actually changed. Re-embedding your whole corpus on every edit is slow and expensive; re-embedding the 3 chunks that changed is cheap. An efficient ingestion job is one that does the least re-embedding necessary.

Deletes: Harder Than They Look (Tombstones)

Deleting a document must remove its vectors — both for correctness (don't retrieve deleted content) and compliance (GDPR's right to be forgotten, data-retention rules). But here's the catch: graph indexes like HNSW can't cheaply remove a node — yanking it out would break the navigation links that pass through it, and a true hard delete means rebuilding the index.

So most vector DBs use a tombstone (soft delete): mark the vector as invalid, filter it out of results immediately, and reclaim the physical space later via a background compaction process (which rewrites the valid vectors into fresh storage and discards the dead ones).

The implication you must know for compliance: a soft-deleted vector may physically linger until compaction runs. For most apps that's fine (it's never returned). For strict right-to-be-forgotten requirements, check whether your DB offers (or you need to trigger) a hard delete + re-index, and verify the data is actually gone — "hidden from results" isn't always "erased from disk."

Keeping It Healthy: Re-indexing & Compaction

Indexes degrade under heavy churn, and you maintain them like any production system:

HNSW graphs fragment as tombstones pile up — recall drops and memory bloats. IVF centroids drift as your data distribution shifts over time (the k-means clusters no longer fit the data well).
The cure is periodic re-indexing / compaction: rebuild or optimize the index to restore recall and reclaim space.
A full HNSW rebuild can make the index unavailable while it runs — so production teams use blue-green: keep serving queries from a static copy of the old index while the new one builds, then swap. (Modern DBs like Qdrant do much of this incrementally, avoiding big rebuilds.)

The good news: a managed database handles compaction and re-indexing for you. On a self-hosted DB, treat it as scheduled maintenance — and monitor recall over time, because silent degradation is the failure mode.

Scaling: When One Machine Isn't Enough

Vector search is memory-bound — the vectors and the ANN index want to live in RAM for speed, so memory is the bottleneck. Your levers, roughly in the order you'd reach for them:

Compress — quantization (the previous section): fit far more vectors per machine. Usually the first and cheapest move.
Shard (horizontal scaling) — split the vectors across multiple nodes; a query hits all shards and merges the results. Adds both capacity and throughput.
Replicate — keep copies of each shard for higher QPS (more queries in parallel) and availability (a node dies, a replica serves).
Disk-based (DiskANN) — keep the index on SSD instead of RAM for huge corpora on cheaper hardware (trading some latency).

One more thing to know: consistency. After you upsert a vector, when is it searchable? Some systems are eventually consistent — a short indexing lag before new data appears in results. For real-time use cases (a user uploads a doc and immediately searches it), check your DB's guarantee.

And the shortcut: managed/serverless vector DBs (Pinecone) do all of this for you — sharding, replication, scaling, re-indexing — which is exactly what you're paying them for.

The Operations in Code

Batch upsert (insert and update — deterministic IDs prevent duplicates) and delete by filter, in Qdrant:

import hashlib
from qdrant_client import QdrantClient, models
client = QdrantClient(url="http://localhost:6333")

def vec_id(source, chunk_i):                 # DETERMINISTIC id -> re-ingest UPDATES, not duplicates
    return hashlib.md5(f"{source}:{chunk_i}".encode()).hexdigest()

# INSERT / UPDATE (upsert): embed in BATCHES, then upsert with metadata
vectors = embed(chunks)                      # embed(chunks) -> a BATCH of embeddings
points = [
    models.PointStruct(id=vec_id("policy.md", i), vector=v,
                       payload={"source": "policy.md", "org_id": 42, "chunk": i})
    for i, v in enumerate(vectors)
]
client.upsert(collection_name="docs", points=points)   # same id -> overwrites (update), no dupes

# DELETE all vectors of a removed doc (or a user's data for GDPR) — by FILTER
client.delete(collection_name="docs", points_selector=models.FilterSelector(
    filter=models.Filter(must=[
        models.FieldCondition(key="source", match=models.MatchValue(value="policy.md"))])))

To update a single changed doc: re-embed its chunks and call the same upsert — the deterministic IDs overwrite the old vectors. To re-index a stale doc, you delete-then-upsert, or just upsert (overwrite). Incremental jobs only touch what changed.

🧪 Try It Yourself

Spot the operational bug. For each, name the problem and fix:

"Every time our nightly job re-ingests the docs, the index doubles in size." → ?
"We fixed a wrong policy doc last week, but search still returns the old wording." → ?
"A customer requested deletion under GDPR; the vector is hidden from results but compliance says it's still on disk." → ?
"Recall slowly got worse over 6 months of heavy adds and deletes." → ?

→ 1: random IDs → duplicates — use deterministic IDs + upsert. 2: edited content but didn't re-embed — re-embed the changed chunks (stale-index trap). 3: tombstone, not hard delete — trigger a hard delete + re-index and verify erasure. 4: HNSW fragmentation / centroid drift — schedule re-indexing/compaction. These four bugs cover ~most of production vector-store ops.

Interactive: an operational console for a live vector index. Starting from a healthy 20-document index, the user runs real operations — re-ingest, edit-text-only, re-embed, delete, compaction, six-months-of-churn, and blue-green re-index — with a deterministic-IDs toggle, and watches five live metrics (stored vectors, duplicates, stale docs, tombstones, recall) plus a running action log react. The four operational bugs are felt directly: re-ingesting with random IDs doubles the index (duplicates); editing a doc's text without re-embedding leaves it silently stale; deleting a doc creates a GDPR tombstone that stays on disk until compaction; and heavy churn fragments the HNSW graph so recall rots until a blue-green re-index restores it. A health panel flags each problem and its fix, turning green only when the index is fresh, deduplicated, compacted, and at full recall. The lesson made tangible: a vector index is alive, and keeping it fresh, correct, and affordable — not search — is production RAG. Deterministic simulation.

Mental-Model Corrections

"The index is a static snapshot." No — data lives; you continuously upsert and delete, and the index degrades without maintenance.
"I can edit a document's text in place." The vector was made from the old text — re-embed and upsert, or the index silently serves stale results.
"Delete removes the data immediately." Usually a tombstone (soft delete) — hidden from results now, physically removed at compaction. Matters for GDPR.
"Set up the index once and forget it." HNSW fragments, IVF centroids drift — re-index periodically (blue-green during rebuilds) and monitor recall.
"Scale = buy a bigger box." Memory is the bottleneck: compress, shard, replicate (or go managed). Vertical scaling runs out.

Key Takeaways

A vector index is live, not static — build an ingestion pipeline: chunk → batch-embed → upsert with metadata, using deterministic IDs so re-ingest updates instead of duplicating.
Updates = re-embed the changed chunks (editing content without re-embedding makes the index silently stale); do it incrementally (only what changed).
Deletes are usually tombstones (soft delete now, compaction reclaims space later) — for GDPR/right-to-be-forgotten, confirm a real hard delete.
Maintain it: HNSW fragments and IVF centroids drift under churn → re-index/compact periodically (blue-green to stay available); monitor recall.
Scale by memory: compress (quantize) → shard → replicate → disk, or use a managed/serverless DB that does it all. Mind consistency (indexing lag) for real-time apps.