AI/ML Wiki

Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

4 min read

Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

Authors: Han, Gao, Liu, J. Zhang, S.Q. Zhang Year: 2023

As large language models grew to hundreds of billions of parameters, full fine-tuning became impractical for most practitioners — updating all weights requires as much GPU memory and storage as training from scratch. Parameter-efficient fine-tuning (PEFT) methods address this by keeping the original pre-trained weights frozen and training only a small set of additional or selected parameters. This survey provides the first unified taxonomy of the PEFT landscape, covering additive, selective, reparameterization, and hybrid methods, and extends the analysis beyond NLP to vision, multimodal, and diffusion models.

Core Idea

PEFT methods share a common insight: the information needed to adapt a model to a new task lives in a much lower-dimensional space than the full weight space. Different families of methods exploit this in different ways — by adding small new modules, by choosing a sparse subset of existing weights, or by reparameterizing updates as compact mathematical structures.

Key Concepts

  • Additive methods: introduce new trainable parameters without changing the frozen backbone. Adapter layers insert bottleneck modules (down-project → nonlinearity → up-project) sequentially inside each Transformer block. Soft prompt tuning prepends learnable continuous embeddings to the input. Prefix tuning inserts trainable key-value pairs directly into each attention layer’s computation.
  • Selective methods: fine-tune a hand-picked or learned subset of the original weights. BitFit trains only bias terms (<0.1% of parameters) and achieves surprisingly competitive NLU performance. Diff pruning learns a sparse task-specific delta over the full parameter space.
  • Reparameterization methods: constrain the weight update to a low-dimensional form. LoRA (Hu et al. 2021) decomposes ΔW = BA with rank r ≪ min(d,k), enabling zero inference overhead after merging. AdaLoRA allocates rank budget adaptively per layer via SVD. DoRA separates magnitude and direction components of the update.
  • Hybrid methods: combine multiple families, e.g. UniPELT gates LoRA, adapters, and prefix tuning simultaneously.
  • Cross-modal PEFT: Visual Prompt Tuning (VPT) applies prompt-style adaptation to ViT; DreamBooth-LoRA applies LoRA to cross-attention layers of Stable Diffusion for personalized image generation.
  • QLoRA: quantizes the frozen base model to 4-bit NormalFloat while keeping LoRA adapters in BF16, enabling 65B-parameter fine-tuning on a single 48 GB GPU.

Results

The survey’s comparative analysis surfaces several consistent findings. The LoRA family dominates reparameterization methods due to zero inference overhead and strong performance across generation tasks. Soft prompt tuning scales favorably with model size — it is most competitive on models above 10B parameters — but underperforms on smaller models. Adapters introduce modest latency (1–20% depending on batch size and layer placement) but remain competitive for NLU and multi-task settings. BitFit is the most extreme form of selective fine-tuning and proves surprisingly effective for classification tasks with under 0.1% trainable parameters. No single PEFT method dominates all task-model combinations: NLU tasks often favor BitFit or LoRA, while generation tasks tend to benefit from prefix tuning or adapter insertion.

Significance

This survey arrived at a pivotal moment when the practitioner community was overwhelmed by a proliferation of PEFT papers with inconsistent terminology and benchmarks. By providing a clean taxonomy and cross-domain coverage, it became a standard reference for orienting new research contributions relative to prior work. The survey’s extension to vision and diffusion models also foreshadowed the rapid adoption of LoRA in image generation workflows (Stable Diffusion community). Its coverage of QLoRA highlights the interaction between quantization and PEFT — a theme that became central to the democratization of LLM fine-tuning through 2024.

Graph View

Backlinks