RAG — Retrieval-Augmented Generation
Lewis et al., Facebook AI Research, NeurIPS 2020
Combines parametric memory (language model) with non-parametric memory (dense retrieval) to answer knowledge-intensive questions without baking facts into weights.
Core Idea
Language models store knowledge in weights (parametric memory), which is static and expensive to update. RAG instead retrieves relevant documents at inference time from a dense index, giving the generator access to up-to-date, verifiable external knowledge.
Query → DPR Retriever → top-k docs → BART Generator → Answer
Architecture
Retriever: Dense Passage Retrieval (DPR)
- Bi-encoder: and produce dense vectors
- Retrieve by maximum inner product search:
- Pre-built FAISS index over all Wikipedia passages
Generator: BART-large
- Seq2seq encoder-decoder
- Conditioned on concatenation of query and retrieved passage(s)
Two Variants
| Variant | Document use | Marginalization |
|---|---|---|
| RAG-Sequence | Same doc for full output | Beam search per doc, re-rank |
| RAG-Token | Different docs per token | Per-token marginalization |
RAG-Token is more flexible; RAG-Sequence works better for multi-word entity answers.
Results
| Task | Performance |
|---|---|
| Natural Questions (Open) | 44.5 EM |
| TriviaQA (Open) | 56.8 EM |
| FEVER | 86.5% accuracy |
| Outperforms T5-11B (closed-book) | with far fewer parameters |
Relationship to Other Work
| Paper | Relationship |
|---|---|
| Vaswani 2017 (Transformer) | BART generator uses Transformer architecture |
| Devlin 2019 (BERT) | DPR encoders initialized from BERT |
| Brown 2020 (GPT-3) | RAG is an alternative to scaling — knowledge via retrieval, not parameters |
| Edge 2024 (GraphRAG) | GraphRAG extends RAG with structured knowledge graphs |
Why RAG Matters
| Problem with LLMs | RAG’s solution |
|---|---|
| Knowledge cutoff | Updatable retrieval index |
| Hallucination | Grounded in retrieved text |
| Opaque reasoning | Retrievals are inspectable |
| Parameter inefficiency | Knowledge in index, not weights |
RAG is now the default architecture for production LLM applications, chatbots, enterprise search, and question-answering systems.