Source — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Source — Retrieval-Augmented Generation (RAG)

Authors: Patrick Lewis et al.
Year: 2020
Venue: NeurIPS 2020 (Facebook AI Research, UCL)

---

Contributions

1. RAG framework: Combines a parametric memory (seq2seq model) with a non-parametric memory (dense retrieval over Wikipedia), enabling knowledge-intensive NLP without retraining.
2. End-to-end training: Both retriever (DPR) and generator (BART) are trained jointly.
3. Two RAG variants: RAG-Sequence (same document for entire answer) and RAG-Token (different documents per token).

---

Methodology

Architecture

• Retriever: Dense Passage Retrieval (DPR) — bi-encoder that maps query and passages to dense vectors, retrieves top-$k$ documents via MIPS (Maximum Inner Product Search) over indexed Wikipedia.
• Generator: BART-large — seq2seq model that generates the answer conditioned on the query and retrieved documents.

RAG-Sequence

Uses one document for the entire output sequence:

$p_{\text{RAG-Seq}}(y|x)\approx {\sum }_{z\in \text{top-}k}{p}_{\eta }(z|x){\prod }_i{p}_{\theta }(y_i|x,z,y_{1:i-1})$

RAG-Token

Can use different documents per output token:

$p_{\text{RAG-Token}}(y|x)\approx {\prod }_i{\sum }_{z\in \text{top-}k}{p}_{\eta }(z|x)p_{\theta }(y_i|x,z,y_{1:i-1})$

Training

• Retriever parameters frozen (DPR pre-trained on Natural Questions)
• Or joint training: update document encoder and generator together
• Marginalize over retrieved documents during training

---

Results

• State-of-the-art on Open-Domain QA (Natural Questions, TriviaQA, WebQuestions)
• Strong performance on fact verification (FEVER), knowledge-grounded dialogue (Wizard of Wikipedia)
• Outperforms closed-book T5-11B on several benchmarks despite smaller parameter count
• Generates more factual, specific, and diverse responses than seq2seq baselines

---

Limitations

• Retrieval quality is a bottleneck — errors in retrieval compound at generation
• Static Wikipedia index: knowledge cutoff problem
• Inference latency: retrieval adds overhead vs. pure parametric models
• Not well-suited for tasks requiring deep multi-step reasoning

---

Significance

RAG became the dominant paradigm for grounding LLMs in external knowledge, forming the basis of virtually all production RAG systems. The separation of parametric (weights) and non-parametric (retrieval) memory is a foundational architecture for LLM applications.