AI/ML Wiki

LoRA: Low-Rank Adaptation of Large Language Models

4 min read

LoRA: Low-Rank Adaptation of Large Language Models

Contributions

  • Low-rank weight decomposition for adaptation: proposes freezing pre-trained model weights and injecting trainable rank-decomposition matrices into each Transformer layer, drastically reducing the number of trainable parameters.
  • No inference latency overhead: unlike adapter layers, LoRA’s trained matrices can be merged back into the original weight matrices at deployment, yielding zero extra latency compared to a fully fine-tuned model.
  • Parameter reduction by 10,000×: for GPT-3 175B, LoRA reduces trainable parameters from 175B to ~4.7M (rank r=4 applied to attention weight matrices), cutting GPU memory by 3× vs. Adam fine-tuning.
  • Empirical validation across domains: demonstrates competitive or superior performance compared to full fine-tuning on RoBERTa, DeBERTa, GPT-2, and GPT-3 across NLU and NLG benchmarks.

Methodology

Motivation

Full fine-tuning of large pre-trained models is expensive: all parameters are updated, requiring optimizer states for every parameter. Prior approaches like adapter layers add sequential modules that increase inference latency. Prefix-tuning and prompt-tuning reduce parameters but suffer from optimization instability and reduced usable sequence length.

Core Idea: Low-Rank Decomposition

LoRA is motivated by the hypothesis that the change in weights during model adaptation has a low intrinsic rank. For a pre-trained weight matrix W₀ ∈ ℝ^(d×k), the update is constrained:

W₀ + ΔW = W₀ + BA

where B ∈ ℝ^(d×r) and A ∈ ℝ^(r×k) with rank r << min(d, k).

  • A is initialized with random Gaussian; B is initialized to zero — ensuring ΔW = 0 at training start, so the model starts from the pre-trained checkpoint.
  • The forward pass becomes: h = W₀x + BAx, scaled by α/r where α is a hyperparameter.
  • W₀ is frozen; only A and B are trained.

Where LoRA Is Applied

In the Transformer, LoRA is applied to the attention weight matrices W_Q, W_K, W_V, and W_O. The paper finds applying to W_Q and W_V sufficient in practice. MLP layers are not adapted in the main experiments.

Merging at Inference

At deployment: W = W₀ + BA. This is a single matrix multiply — no added latency. LoRA can be “unmerged” to quickly switch between tasks by subtracting BA and adding a different B’A’.

Relationship to Other Methods

  • Adapter layers: add sequential bottleneck modules; increase inference latency by 20–30% in low-batch, latency-sensitive settings. LoRA has zero overhead.
  • Prefix tuning: optimizes a soft prompt prepended to the input; reduces usable sequence length and has a non-monotonic effect on performance with rank.
  • Full fine-tuning: special case of LoRA when r = d (full rank).

Results

NLU — GLUE (RoBERTa-large, DeBERTa-XXL)

  • LoRA matches or exceeds full fine-tuning on GLUE benchmarks using only 0.3% of the parameters of RoBERTa-large.
  • DeBERTa-XXL with LoRA achieves 91.6 average GLUE score, competitive with the full fine-tuned model (91.4).

NLG — GPT-2 (E2E, WebNLG, DART)

  • On E2E NLG, LoRA (r=4) achieves 70.4 BLEU vs. 68.2 for fine-tuning and 69.7 for best adapter variant.
  • Outperforms prefix-tuning and adapter baselines while using far fewer parameters.

GPT-3 175B (WikiSQL, MultiNLI, SAMSum)

  • LoRA with r=4 achieves 73.4% on WikiSQL vs. 73.8% for fine-tuning, using only ~4.7M trainable parameters (vs. 175B).
  • Reduces checkpoint storage by 10,000× — critical for serving multiple adapted models.

Ablations

  • Rank r=1 to 4 suffices for most tasks; increasing beyond 8 yields marginal gains.
  • Applying LoRA to more weight matrices (W_Q, W_K, W_V, W_O simultaneously) often outperforms applying to a single matrix with proportionally higher rank.

Limitations

  • The choice of rank r is a hyperparameter requiring tuning; too low a rank may miss task-relevant information.
  • LoRA does not adapt MLP layers or layer norms; for some tasks, this may be suboptimal.
  • The assumption of low intrinsic rank is empirically validated but not theoretically proven for all tasks or domains.
  • Merging at inference means dynamic task switching requires re-merging matrices, adding per-switch overhead.

Graph View

Backlinks