Knowledge Graphs for Retrieval (Neo4j/Memgraph)

Introduction

Welcome to the final section of the RAG container — the techniques for the questions that everything so far still can't answer well. The entire advanced-retrieval stack you just built — hybrid, fusion, reranking, transforms, routing — is fundamentally about one thing: finding the passages most similar to a query. That's incredibly powerful. It's also structurally blind to relationships.

Ask "Which of our suppliers are owned by companies we're also competing with?" and similarity search flounders — not because the facts are missing, but because the answer lives in how entities connect, scattered across many documents, and vector search treats every chunk as an isolated island. This lesson introduces the tool built for exactly this: the knowledge graph.

In this lesson you'll learn:

Why vector search can't connect the dots — and graphs can
The triple — the atom of a graph — and querying by traversal (Cypher)
How to build a knowledge graph from text with an LLM (Neo4j / Memgraph)
The honest cost — and the 2026 hybrid consensus (vector + graph)

An infographic titled 'Knowledge Graphs — Retrieval by Relationship, Not Similarity'. Two contrast cards. Vector RAG (retrieval by similarity): finds passages ABOUT your query, returns the top-k isolated chunks with no idea how they connect, so multi-hop and connect-the-dots questions slip through. Knowledge-Graph RAG (retrieval by relationship): finds what your query is CONNECTED to by traversing edges N hops deep, answering questions whose answer lives in the relationships between facts, not in any one passage. A strip on the atom of a graph: the (entity)-[relationship]->(entity) triple, with an example chain Alice-WORKS_AT->Acme-ACQUIRED->Beta-SUPPLIES->Gamma; nodes are entities, edges are typed relationships, queried by traversal in Cypher. Two cards: Build it from text — an LLM extracts entities plus relationships into triples stored in a graph DB, keeping both a lexical graph (chunks) and an entity graph, using LLMGraphTransformer or PropertyGraphIndex, stored in Neo4j or Memgraph. When graphs win — multi-hop reasoning, aggregation over relationships, explainable traceable paths, and slow-changing relational domains like org charts, supply chains, regulations, biomedicine. A caveat: not free — building a graph costs 10 to 100 times more than vector indexing (millions of LLM tokens), extraction errors propagate, and graphs go stale (ontologies drift) without ownership; it's a complement to vector RAG, not a replacement; the 2026 consensus is vectors for the entry point, graphs for relational depth. Banner: when the answer lives in the connections between facts, not in any single passage, use a graph; vectors for aboutness, graphs for relationships.

Similarity vs. Relationships: Two Different Questions

There are two fundamentally different things you can ask of a knowledge base, and they need different machinery:

"Find me things about X." — aboutness / similarity. This is vector search: embed the query, return the top-k most semantically similar chunks. Perfect for "what does our policy say about refunds?"

"Find me things connected to X." — relationships / structure. This is graph search: traverse the edges between entities, however many hops it takes. Required for "who reports to managers who joined before 2020?" or "how is Alice connected to Gamma?"

Vector search has a hard ceiling here, and it's worth being precise about why: it returns the top-k independent chunks and has no representation of the fact that an entity in chunk A is the same as — or related to — an entity in chunk B. Relationships are severed at chunking. You can retrieve the chunk about Acme and the chunk about Bob, but nothing in either says "Bob works at a company Acme acquired" — that fact exists only in the connection between them, which a pile of similar chunks can't represent. Multi-hop questions are where vector RAG quietly fails.

The Knowledge Graph: Entities, Relationships, Triples

A knowledge graph represents knowledge as a network: nodes are entities (a person, a company, a product, a gene), and edges are typed relationships between them (WORKS_AT, ACQUIRED, IS_A_COMPONENT_OF). Both can carry properties (key-value attributes) — this is the property graph model used by Neo4j and Memgraph.

The fundamental unit is the triple: (subject) —[predicate]→ (object) — e.g. (Alice)-[:WORKS_AT]->(Acme). A whole graph is just many triples sharing nodes, which is what lets you chain them:

(Alice)-[:WORKS_AT]->(Acme)-[:ACQUIRED]->(Beta)-[:SUPPLIES]->(Gamma)

Now "how is Alice connected to Gamma?" has a precise, traceable answer — follow the chain — even though no single document ever states it. You query a graph by traversal, using a graph query language; the dominant one is Cypher (Neo4j, Memgraph):

// A property graph: nodes (entities) + typed relationships + properties.
CREATE (alice:Person {name:'Alice'})-[:WORKS_AT]->(acme:Company {name:'Acme'})
CREATE (bob:Person {name:'Bob'})-[:WORKS_AT]->(beta:Company {name:'Beta'})
CREATE (acme)-[:ACQUIRED]->(beta)
CREATE (beta)-[:SUPPLIES]->(gamma:Company {name:'Gamma'});

// MULTI-HOP question — impossible for plain vector search, trivial for a graph:
// "Who works at a company Acme acquired?"
MATCH (:Company {name:'Acme'})-[:ACQUIRED]->(c)<-[:WORKS_AT]-(p:Person)
RETURN p.name;                       // → "Bob"  (Acme→ACQUIRED→Beta←WORKS_AT←Bob)

// "How is Alice connected to Gamma?" — traverse 1..3 hops, return the PATH:
MATCH path = (:Person {name:'Alice'})-[*1..3]-(:Company {name:'Gamma'})
RETURN path;                         // Alice→Acme→Beta→Gamma — explainable & exact

Notice what graph retrieval gives you that similarity can't: it's exact (not approximate), multi-hop (follow relationships as far as needed), and explainable (the returned path literally is the reasoning). This is the killer property for auditable domains — you can show why two things are connected.

See It: Traverse What Vector Search Can't

This is the whole idea in one widget. Below is a small property graph; ask a relationship question and watch the traversal path light up — with the Cypher query, the answer, and exactly why pure similarity search would miss it:

Ask a relationship question and watch the traversal path light up across the graph — with the Cypher query, the answer, and exactly why vector search can't connect those dots.

Try the 3-hop question ("How is Alice connected to Gamma?"). The answer is a chain — Alice → Acme → Beta → Gamma — that appears in no single chunk and has no semantic similarity to the question's wording. A graph finds it by following relationships; a vector index has no relationships to follow. That gap is the entire reason knowledge graphs exist in RAG.

Building a Knowledge Graph From Text

Most of your data isn't a graph — it's unstructured documents. So the central engineering task is constructing the graph from text, and in 2026 that's done with an LLM as the graph builder: prompt it (often with a target schema) to read each chunk and emit the entities and relationships it contains as structured triples.

from langchain_experimental.graph_transformers import LLMGraphTransformer
from langchain_neo4j import Neo4jGraph
from langchain_openai import ChatOpenAI

# An LLM reads each chunk and EXTRACTS (entity)-[relationship]->(entity) triples.
transformer = LLMGraphTransformer(llm=ChatOpenAI(model="gpt-5.5"))
graph_docs = transformer.convert_to_graph_documents(chunks)   # nodes + relationships

graph = Neo4jGraph()
graph.add_graph_documents(
    graph_docs,
    include_source=True,        # link each entity back to its ORIGIN chunk
    baseEntityLabel=True,       # → keeps BOTH a "lexical" graph (chunks) and an "entity" graph
)
# ⚠️ Extraction is imperfect (missed/wrong triples) and costs an LLM call PER chunk —
#    building the graph can be 10–100× the cost of plain vector indexing.

The tooling is mature: LangChain's LLMGraphTransformer (contributed by Neo4j) and LlamaIndex's PropertyGraphIndex (with extractors like SchemaLLMPathExtractor) both do this. A crucial detail in modern KG-RAG: you keep two linked graphs — a lexical graph (the original chunks, with embeddings) and an entity graph (the extracted nodes/relationships), wired together. That way you can enter via vector search and traverse via relationships — which is exactly the hybrid pattern we'll get to.

But be clear-eyed: this extraction step is the hard, expensive, error-prone part. It requires entity recognition, relation extraction, and entity disambiguation (is "J. Smith" the same node as "John Smith"?), and the graph is only as good as what the LLM extracts. Errors here propagate through every reasoning chain that touches those nodes.

Graph Databases & the Honest Cost

You store and query the graph in a graph database. The main players in 2026:

Neo4j — the dominant property-graph database; mature Cypher + a deep GenAI/GraphRAG ecosystem. The default choice.
Memgraph — in-memory, high-performance, Cypher-compatible; strong for real-time graphs.
Also: Amazon Neptune, ArangoDB / Dgraph / NebulaGraph (multi-model), and embedded options (Kùzu, FalkorDB).

Now the honest part — because knowledge graphs are often oversold. They are dramatically more expensive and complex than vector RAG:

Construction cost: ~10–100× a vector index. A ~100k-chunk corpus can burn millions of LLM tokens and take hours to days to build (an LLM extraction call per chunk).
System complexity: you now operate a graph DB (and usually a vector DB too), an extraction pipeline, and a schema — far more than "put docs in a vector store."
Maintenance & staleness: ontologies drift, entities get renamed, relationships change. Graphs go stale faster than chunks without a dedicated owner — it's an ongoing commitment, not a one-time build.

So only reach for a graph when relationships genuinely are the answer: multi-hop reasoning, aggregation over relationships, hard explainability/auditability needs — and ideally over a slow-changing relational domain (regulations, technical manuals, org/supply-chain/biomedical data), not a real-time feed. For "find passages about X," a graph is expensive overkill.

The 2026 Consensus: Hybrid (Vector + Graph)

Here's the resolution the field has converged on, and it should sound familiar from the hybrid-search lesson: it's not graph vs vector — it's both. The community pattern for 2026 is:

Vectors for the entry point (semantic search to find the relevant entities/chunks), graphs for relational depth (traverse from there to connect the dots).

A typical HybridRAG flow: use vector search to find the entities most relevant to the query, then traverse the graph from those entities to gather connected context, and feed the combined result to the LLM. Two ways to build it:

Single-engine: store embeddings as node properties and use the graph DB's own vector index + traversal (Neo4j, Memgraph, Arango all support this) — fewer moving parts, simpler consistency.
Two systems: a dedicated vector DB for ANN entry + a graph DB for traversal.

The payoff is real where relationships matter: Microsoft's GraphRAG reported +26% answer comprehensiveness and +57% response diversity over standard vector retrieval (and better faithfulness, since explicit paths preserve relational context); an AWS test saw up to +35% precision from adding graph structure. That's the bridge to the next lesson — GraphRAG — which builds the graph, clusters it into communities, and summarizes them to answer global questions like "what are the main themes across this whole corpus?" that neither plain vector search nor simple traversal can.

🧪 Try It Yourself

Drive the graph widget, then reason about when to reach for one:

In the interactive, run the 3-hop Alice→Gamma question. Why is this answer fundamentally invisible to a vector search, no matter how good the embedding model?
Write the triple(s) for: "Dr. Lee authored the 2021 study, which was funded by the NIH." What are the nodes, edges, and any properties?
For each, choose vector or graph (or both): (a) "What does the handbook say about parental leave?" (b) "Which board members also sit on the boards of our competitors?" (c) "Summarize the refund policy." (d) "Trace every component affected if part #A12 is recalled."
Your team wants to graph a real-time news feed for a Q&A bot. Based on the honest tradeoffs, what's your concern?
You build a HybridRAG system. What job does the vector half do, and what job does the graph half do?

→ (1) The answer is a chain of relationships (Alice-WORKS_AT-Acme-ACQUIRED-Beta-SUPPLIES-Gamma) that appears in no single chunk and shares no wording with the question — there's nothing for similarity to match; only traversal finds it. (2) Nodes: (Dr. Lee:Person), (2021 Study:Publication {year:2021}), (NIH:Organization); edges: (Dr. Lee)-[:AUTHORED]->(Study), (NIH)-[:FUNDED]->(Study). (3a) Vector (aboutness). (3b) Graph (multi-hop relationships). (3c) Vector. (3d) Graph (traverse the component/dependency tree). (4) Concern: graphs are expensive to build and go stale fast; a real-time feed means constant re-extraction — a poor fit (graphs suit slow-changing domains). (5) Vector = entry point (semantically find the relevant entities/chunks); graph = relational depth (traverse from those entities to connect the dots).

Mental-Model Corrections

"Vector search can answer any question if I retrieve enough chunks." No — it has no representation of relationships; multi-hop "connect-the-dots" answers live in the links between facts, which similarity over isolated chunks can't reconstruct.
"A knowledge graph is just a fancy vector DB." Different model: a graph stores entities + typed relationships and is queried by traversal (Cypher), not by nearest-neighbour similarity.
"Graphs are strictly better than vectors." They're better for relationships/multi-hop/explainability, worse (and far costlier) for plain aboutness. Use both — vectors for entry, graphs for depth.
"Building the graph is the easy part." It's the hard, expensive (10–100×), error-prone part — LLM extraction + disambiguation, and errors propagate.
"Build a graph once and you're done." Graphs go stale (ontology drift, renamed entities); they need ongoing ownership — worse for fast-changing data.
"Knowledge graphs are for every RAG app." Only when relationships are the answer. For "find passages about X," a graph is expensive overkill.

Key Takeaways

Vector search retrieves by similarity (aboutness); knowledge graphs retrieve by relationship (structure). Multi-hop / connect-the-dots questions live in the links between facts — where vector RAG is structurally blind.
A graph is entities (nodes) + typed relationships (edges) + properties; the atom is the triple (subject)-[predicate]->(object). You query it by traversal with Cypher — exact, multi-hop, and explainable (the path is the reasoning).
Build it from text with an LLM (LLMGraphTransformer, PropertyGraphIndex), keeping a lexical graph (chunks) and an entity graph, in Neo4j / Memgraph. Extraction is the hard, error-prone step.
Honest cost: 10–100× pricier to build than vector indexing, more system complexity, and ongoing staleness. Reach for it only for multi-hop / aggregation / explainability over slow-changing relational domains.
2026 consensus = hybrid: vectors for the entry point, graphs for relational depth (HybridRAG). It complements vector RAG; it doesn't replace it.
Next: GraphRAG — extract a graph, cluster it into communities, and summarize them to answer global questions across an entire corpus that neither vector search nor simple traversal can.