GraphRAG: When Vector Search Falls Short

Where flat vector RAG breaks down

Standard RAG works by a single mechanism: embed the query, find the chunks whose vectors sit nearest to it, hand those chunks to the model. That mechanism is excellent at one thing, retrieving text that resembles the question, and the ceiling of that thing is lower than most teams notice until a specific class of question keeps failing. Three shapes of question expose the ceiling, and they fail for the same underlying reason: similarity is a local signal, and these questions are not local.

The first is the multi-hop question, where the answer is a chain of facts that no single passage states. "Which of our suppliers is exposed to the vendor that had the 2024 breach?" requires linking a breach report to a vendor, that vendor to its subsidiaries, and those subsidiaries to your supplier list. Each link may live in a different document. Vector search retrieves passages similar to the query, but the passage naming your supplier need not resemble the passage naming the breach at all, so the chain is never assembled. The retriever returns plausible neighbors and the model, lacking the connecting facts, guesses.

The second is the entity-centric question: "everything we know about Acme across all sources." An entity of any importance is mentioned in dozens of chunks, and top-k retrieval returns only the k most similar, silently dropping the rest. You get a sample of what looks most like the query phrasing, not a complete view of the entity.

The third, and the hardest, is the holistic question: "what are the recurring themes across all 4,000 incident reports?" No chunk contains the answer, because the answer is a property of the corpus as a whole. There is nothing for similarity to match against. Retrieving the twenty nearest chunks answers a question nobody asked.

What GraphRAG actually is

GraphRAG replaces the assumption that retrieval means ranking chunks by similarity with a different one: that the corpus has a structure worth extracting first. At indexing time, an LLM reads the source documents and pulls out entities (people, organizations, products, systems), the relationships between them, and often the claims made about them. Those extractions become a knowledge graph: nodes are entities, edges are relationships. Retrieval can then traverse that graph, following edges to assemble connected evidence, rather than only fetching the nearest vectors.

Microsoft published a GraphRAG method and an open-source project under that name in 2024, and it is the most cited concrete implementation. It is worth being precise about the category, though: knowledge-graph-augmented retrieval is a family of approaches, not a single product. Microsoft's design is one well-documented point in a space that includes graph databases queried by generated Cypher, entity-linked hybrid retrievers, and lighter schemes that attach a small neighborhood to each chunk. When a vendor says "we do GraphRAG," the first question is which of these they mean.

  FLAT VECTOR RAG          GRAPH-BASED RETRIEVAL

  query                    query
    |                        |
    v                        v
  [embed]                  [find entities]
    |                        |
    v                        v
  nearest k chunks         graph: A --owns--> B
    |                              B --hit-by--> C
    v                        |
  LLM answers              traverse edges + chunks
                             |
                             v
                           LLM answers with the chain
            

The graph is not a replacement for the text; it is an index over it. Nodes and edges carry pointers back to the source chunks, so the model still grounds its answer in real passages. What changes is how those passages are found: by relationship and by community, not by cosine distance alone. That shift is what lets graph retrieval answer the multi-hop and holistic questions that defeat flat RAG, and it is also the source of every cost discussed further down.

Building the graph

The index is where GraphRAG earns both its capability and its bill. The pipeline is several stages deep, and each one is an engineering decision, not a switch you flip. Understanding the stages is the only honest way to reason about what graph retrieval costs and where it goes wrong.

1. Extraction. An LLM processes the corpus chunk by chunk and emits structured output: entities with types, relationships between them, and often claims (assertions an entity makes or that are made about it). This is the dominant cost. Every chunk gets at least one model call, sometimes several, over the entire corpus.
2. Entity resolution. "Acme," "Acme Corp," and "ACME Corporation" are one entity extracted three ways. Deduplication and resolution merge them, or the graph fragments into near-duplicate nodes that defeat traversal. This step is underrated and is where quality is quietly won or lost.
3. Community detection. The Leiden algorithm partitions the graph into communities: densely connected clusters of entities that tend to concern a shared topic. Communities are often computed at several levels, giving a hierarchy from broad themes down to tight sub-topics.
4. Community summarization. An LLM writes a summary of each community. These summaries are precomputed at indexing time and are what makes corpus-wide questions answerable later without re-reading everything.

Two properties fall out of this. First, indexing is heavier and materially more expensive than chunk-and-embed, which is a single embedding pass with no generation. Graph construction is many LLM calls for extraction plus more for summarization. Second, the graph is a derived artifact with its own correctness: a botched extraction or a missed merge is a wrong edge that traversal will faithfully follow to a confident, wrong answer. The graph inherits the extraction model's mistakes.

Local search vs global search

Once the graph exists, GraphRAG offers two distinct query paths, and choosing between them is a routing decision the system makes per question. They are not variants of one retriever; they answer different shapes of question by different mechanisms, and conflating them is a common source of disappointment ("we tried GraphRAG and it was slow on lookups" usually means global search was used where local would have done).

Local search answers entity-focused questions. It anchors on the entities named in the query, pulls their neighborhood from the graph (connected entities, the relationships between them, and the underlying text chunks), and assembles that context for the model. This is the path for "what is Acme's relationship to our top supplier?" It is the multi-hop workhorse: traversal is how the chain of connecting facts gets built.

Global search answers corpus-wide theme questions. Rather than retrieve chunks, it map-reduces over the precomputed community summaries: each relevant summary produces a partial answer (the map step), and those partials are combined into a final response (the reduce step). This is the path for "what are the dominant risk themes across every audit report?" It reads structured summaries of the whole corpus rather than a similarity-selected sample, which is exactly why it can answer the holistic question flat RAG cannot touch.

The practical implication for an architect is that a graph retrieval layer needs a router in front of it: a step that classifies the incoming question and dispatches to local, global, or plain vector search. Ship only one path and half your questions land on the wrong mechanism.

A decision guide

Graph retrieval is a powerful tool for a specific class of questions, not a general upgrade to your RAG stack. The decision turns on two things: the shape of the questions your users actually ask, and whether the corpus is entity-rich enough that extraction produces a useful graph rather than a sparse one. A support knowledge base answering "how do I reset my password" gains nothing from a graph; a corpus of investigations, contracts, or incident reports where the value is in the connections gains a great deal.

The most important framing is that this is rarely either/or. Production systems increasingly combine vector, keyword, and graph retrieval, routing each question to the path that fits and often blending results. The graph handles the connected and holistic questions; vector plus keyword (covered in Hybrid Search & Re-Ranking) handles the majority of straightforward lookups faster and more cheaply. Use the table as a starting heuristic, not a verdict.

The honest default for most enterprises: start with vector plus hybrid retrieval, instrument which questions fail, and add graph retrieval to the corpora and query classes where the failures are demonstrably connection and sensemaking failures. Adopting a graph because it is sophisticated, rather than because your question mix demands it, buys you indexing bills and operational complexity for a capability your users are not exercising.

The cost and operations reality

The capability is real; so is the invoice, and the ROI has to clear a genuine bar. The costs fall in three buckets, and each is easy to underweight during a proof of concept that runs on a small, static corpus and then surprises the team at production scale.

Indexing cost. Building the graph is many LLM calls: extraction over every chunk, then community summarization. On a large corpus this is a substantial, recurring expense, not a one-time toy cost, and it scales with corpus size and entity density rather than with query volume. Global search additionally spends more per query than a vector lookup, because it fans out across community summaries. Budget both the build and the run.

Freshness. A vector index absorbs a new document by embedding and inserting it. A graph must re-extract entities from the new text, resolve them against existing nodes, potentially rewire edges, and, if communities shifted, recompute affected summaries. Incremental update is an active area and the tooling is improving, but graph freshness is genuinely harder than vector freshness. For a corpus that changes hourly, this is a first-order design constraint, not a footnote.

Operational complexity. You now run and debug an extraction pipeline, a graph store, a community-detection step, a summarization step, and a router, on top of the vector stack you probably still keep. When an answer is wrong, the failure could live in extraction, resolution, community assignment, summarization, or routing, which is a longer diagnostic chain than "the retriever ranked the wrong chunk." That surface area is a standing operations cost measured in engineer time.

The architect view

The framing that keeps teams out of trouble is this: graph retrieval is one strategy in the retrieval toolkit, chosen per query class, not a wholesale replacement for vector search. The moment it gets pitched as "the next generation of RAG that makes embeddings obsolete," the project is being sold on category rather than on fit, and the indexing bills and freshness headaches arrive before the promised gains do. Similarity search remains the right, cheap answer for the plurality of enterprise questions. The graph earns its place on the questions similarity cannot reach.

So the sequencing is almost always the same. Most enterprises should start with vector plus hybrid retrieval, get evaluation discipline in place (a golden set, retrieval metrics, a failure taxonomy, as covered in Evaluating RAG), and let the data show which questions fail and why. Where the residual failures cluster around connected and holistic questions over an entity-rich corpus, add a graph path behind a router and measure whether it moves the numbers that matter. That is a decision made with evidence, and it is reversible, because the vector stack stays underneath.

Two closing notes for whoever owns this decision. First, insist on the same evaluation rigor you would apply to any retrieval change: a graph that produces confident, well-connected, wrong answers from a bad extraction is more dangerous than a vector retriever that visibly returns nothing, because the wrongness is fluent. Second, treat "GraphRAG" as a family, not a product. Ask which mechanism a vendor means, what the indexing cost is on your corpus, and how freshness is handled, before it becomes a line item. The teams that get value from graph retrieval are the ones that added it deliberately to a specific question class. The teams that regret it are the ones that adopted it as a default. You can build the entity extraction and see a graph form from your own text in the Knowledge Graph Builder in the Lab.