Source Summary: Edge et al. 2024 — GraphRAG
Full title: From Local to Global: A Graph RAG Approach to Query-Focused Summarization
Authors: Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Jonathan Larson et al.
Affiliation: Microsoft Research
Venue: arXiv: 2404.16130 (April 2024)
PDF: papers/edge-2024-from-local-to-global-a.pdf
Contributions
- GraphRAG — a retrieval-augmented generation pipeline that builds a knowledge graph from source documents and uses community summaries for global question answering.
- Global sensemaking — demonstrates that vector RAG fails on corpus-level questions (“what are the main themes in this dataset?”) while GraphRAG succeeds.
- LLM-as-judge evaluation — a methodology for evaluating RAG systems on open-ended questions without ground-truth answers; using one LLM to generate questions and another to judge answers.
- Community detection for summarization — hierarchical Leiden algorithm on the entity graph produces community structures; LLM-generated summaries at each level enable multi-granularity QA.
The Problem with Vector RAG
Standard RAG: embed documents → embed query → retrieve top-k similar chunks → generate answer.
This works for local questions (specific fact lookup). It fails for global sensemaking questions:
Local: "When was the Treaty of Versailles signed?" → one chunk answers this
Global: "What are the main themes in this historical corpus?" → no single chunk answers this;
requires synthesis across the entire dataset
Vector RAG cannot aggregate information across many documents in a principled way. It retrieves fragments and hopes the LLM synthesizes them — but for large corpora, the relevant information is spread too thinly.
The GraphRAG Pipeline
Phase 1: Knowledge graph construction (indexing)
Source documents
↓ LLM extraction
Entity knowledge graph:
nodes = key entities (people, places, concepts, events)
edges = relationships between entities
↓ Community detection (Leiden algorithm)
Hierarchical community structure:
Level 0: fine-grained clusters (~5-10 entities)
Level 1: medium clusters
Level 2+: coarse clusters (major themes)
↓ LLM summarization
Community summaries at each level
Phase 2: Query answering (retrieval)
Query
↓ Map step (parallel)
Each community summary → partial answer to query
↓ Reduce step
All partial answers → final synthesized answer
Key Results
Evaluated on two corpora (~1M tokens each): news articles and a podcast transcript.
Global questions (e.g., “What are the most important topics in this corpus?“):
| Method | Comprehensiveness | Diversity |
|---|---|---|
| Vector RAG | baseline | baseline |
| GraphRAG (C0 communities) | +16% | +18% |
| GraphRAG (C2 communities) | +23% | +27% |
Local questions: GraphRAG performs comparably to vector RAG, with no degradation.
Why This Matters
GraphRAG demonstrates a concrete performance advantage of structured (graph) over unstructured (vector) knowledge representation for a class of real tasks. This is the practical validation of the symbolic AI insight that structure enables reasoning:
- Entities and relationships (nodes and edges) are the symbolic layer
- Community detection provides automatic ontology construction
- Hierarchical summaries enable reasoning at multiple levels of abstraction
The pipeline is also a demonstration of hybrid symbolic-neural architecture: LLMs extract and summarize (neural), graph algorithms structure and traverse (symbolic).
Limitations
- Knowledge graph construction requires significant LLM compute (one extraction pass per document)
- Graph quality depends on LLM extraction quality — errors propagate
- Community detection is domain-agnostic; may not reflect domain-relevant clustering
- Local query performance does not exceed standard RAG significantly
Key References (from paper)
- Lewis et al. 2020 — original RAG paper
- Baumel et al. 2018 — query-focused summarization
- Traag et al. 2019 — Leiden community detection algorithm
- Achiam et al. 2023 — GPT-4 (LLM used in experiments)