AI/ML Wiki

LoRA: Low-Rank Adaptation of Large Language Models

4 min read

LoRA: Low-Rank Adaptation of Large Language Models

Authors: Hu, Shen, Wallis, Allen-Zhu, Li, S. Wang, L. Wang, Chen (Microsoft) Year: 2021

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning (PEFT) method that makes it practical to adapt enormous pre-trained language models to new tasks without updating all of their weights. Instead of modifying the original weight matrices, LoRA freezes them and injects small, trainable low-rank matrices alongside each linear layer in the Transformer. The resulting adapted model performs comparably to a fully fine-tuned one but requires orders of magnitude fewer trainable parameters — for GPT-3 175B, from 175 billion down to roughly 4.7 million.

Core Idea

The central hypothesis is that the weight updates needed for task adaptation lie in a low-dimensional subspace — i.e., the “intrinsic rank” of the adaptation is small. For a frozen weight matrix W₀ ∈ ℝ^(d×k), LoRA parameterizes the update as:

ΔW = B · A    where B ∈ ℝ^(d×r), A ∈ ℝ^(r×k), r << min(d, k)

During training, W₀ is fixed and only B and A are updated. B is initialized to zero and A to random Gaussian, so at initialization ΔW = 0 and training starts exactly from the pre-trained checkpoint. A scaling factor α/r controls the magnitude of the update.

Crucially, at inference time B·A can be added directly to W₀ — a one-time matrix addition — so the deployed model has no extra layers and no added latency compared to the original.

Key Concepts

  • Low-rank decomposition: expresses the weight update ΔW as a product of two thin matrices, limiting the degrees of freedom to r·(d+k) rather than d·k.
  • Frozen base weights: the original model’s knowledge is preserved exactly; only the small adapter matrices change.
  • Zero-overhead inference: LoRA matrices are merged before deployment, unlike adapter layers which add sequential modules and increase latency.
  • Rank r: the key hyperparameter. r=1 to 8 suffices for most benchmarks; larger r rarely helps.
  • Target modules: typically applied to W_Q and W_V (query and value projection) of each attention layer; applying to all four attention matrices (Q, K, V, O) provides modest additional gains.
  • Multi-task switching: a single base model can be paired with multiple LoRA adapters (one per task), switched by subtracting one BA and adding another — enabling efficient multi-task serving.

Results

On RoBERTa-large, LoRA with ~0.3% of the trainable parameters matches full fine-tuning across GLUE benchmarks. On GPT-2 NLG tasks (E2E, WebNLG, DART), LoRA outperforms adapter baselines while using fewer parameters. Most impressively, on GPT-3 175B, LoRA achieves near-parity with fine-tuning on WikiSQL and MultiNLI while shrinking the per-task checkpoint from 350 GB to under 35 MB — a 10,000× reduction.

Ablation studies show that distributing the rank budget across multiple weight matrices (e.g., r=2 each for W_Q and W_V) consistently beats concentrating all rank in a single matrix.

Significance

LoRA became the dominant approach for fine-tuning large language models in practice. Its zero inference overhead differentiates it from adapter-based methods, and its simplicity makes it easy to integrate into existing training pipelines. The technique directly enabled the proliferation of task-specific and instruction-tuned variants of open-source models (LLaMA, Mistral, etc.) by reducing the compute and storage costs of adaptation to levels accessible on consumer hardware.

LoRA also influenced subsequent PEFT research: QLoRA quantizes the frozen base model to 4-bit precision while keeping LoRA adapters in higher precision, enabling fine-tuning of 65B-parameter models on a single 48 GB GPU. DoRA (Weight-Decomposed LoRA) further decomposes the update into magnitude and direction components.

Graph View

Backlinks