Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

Contributions

• Unified taxonomy of PEFT methods: organizes the fragmented PEFT literature into a coherent framework covering additive methods (adapters, soft prompts), selective methods (sparse masking), reparameterization methods (LoRA family), and hybrid approaches.
• Systematic comparison across modalities: surveys PEFT not only for NLP but also for vision models, multimodal models, and diffusion models — providing a cross-domain perspective rarely seen in prior surveys.
• Analysis of efficiency-performance trade-offs: quantifies the relationship between the number of trainable parameters, memory footprint, and downstream task performance across dozens of methods and benchmarks.
• Coverage of practical deployment considerations: discusses adapter merging, multi-task inference, quantization-aware PEFT (QLoRA), and hardware-aware design.

Methodology

This is a survey paper. Its methodology consists of systematic literature review, categorization, and comparative analysis rather than novel algorithm development.

Taxonomy of PEFT Methods

1. Additive Methods — add new trainable parameters while freezing the original model:

• Adapter layers: insert small bottleneck modules (down-project → nonlinearity → up-project) within each Transformer layer. Introduced by Houlsby et al. (2019). Sequential placement adds inference latency.
• Parallel adapters: place the adapter in parallel with the FFN or attention sublayer to avoid sequential overhead.
• Soft prompt tuning: prepend trainable continuous token embeddings to the input sequence (Lester et al. 2021). No architectural changes; only input embeddings are optimized.
• Prefix tuning: insert trainable key-value pairs into the attention computation of every layer (Li & Liang 2021). More expressive than prompt tuning but reduces usable context length.

2. Selective Methods — fine-tune a chosen subset of the original parameters:

• Diff pruning: learn a sparse task-specific parameter delta; only a small fraction of weights change.
• BitFit: fine-tune only bias terms — surprisingly competitive on many NLU tasks with <0.1% trainable parameters.
• Fisher-based selection: use Fisher information to identify the most task-relevant parameters.

3. Reparameterization Methods — express weight updates in a lower-dimensional form:

• LoRA (Hu et al. 2021): decompose ΔW = BA; zero inference overhead after merging.
• AdaLoRA: allocate rank budget adaptively across layers and weight matrices using singular value decomposition, pruning less important singular components.
• KronA: use Kronecker product decomposition instead of low-rank matrices.
• DoRA: decompose pre-trained weights into magnitude and direction, fine-tuning each separately.

4. Hybrid Methods — combine additive and reparameterization approaches:

• UniPELT: combines prefix tuning, LoRA, and adapters with gating mechanisms.
• LLaMA-Adapter: adds learnable prompts with zero-initialized attention gates for instruction following.

Cross-Domain Coverage

The survey extends PEFT analysis to:

• Vision Transformers (ViT): Visual Prompt Tuning (VPT) adds trainable tokens to each ViT layer.
• Multimodal models: surveys adaptation of CLIP, ALIGN, Flamingo.
• Diffusion models: LoRA applied to cross-attention layers of Stable Diffusion (DreamBooth-LoRA).
• Speech models: adapter-based fine-tuning of Whisper.

Results

Key Findings from Comparative Analysis

• LoRA family dominates in the reparameterization category: zero inference overhead and strong performance make it the most widely adopted approach.
• Soft prompt tuning scales with model size: effective for very large models (>10B parameters) but underperforms full fine-tuning on smaller models.
• Adapters have consistent but modest overhead: ~1–3% latency increase in typical batch inference settings.
• Optimal PEFT method is task- and model-dependent: no single method wins across all settings; NLU tasks often favor BitFit or LoRA; generation tasks benefit from prefix tuning or full adapter insertion.
• QLoRA (quantized LoRA) achieves fine-tuning of 65B-parameter models on a single GPU, enabling research-scale work on consumer hardware.

Parameter Efficiency Summary (approximate)

Method	Trainable %	Latency Overhead	Best Use Case
Full fine-tuning	100%	0%	Maximum performance
Adapter	0.5–3%	2–20%	NLU, multi-task
LoRA (r=8)	0.1–1%	0%	Generation, large models
Prompt tuning	<0.01%	0%	Inference-only adaptation
BitFit	<0.1%	0%	NLU, classification

Limitations

• As a survey, it does not propose new methods and its comparisons are constrained by heterogeneous reported benchmarks across primary papers.
• Coverage is inevitably incomplete given the rapid pace of PEFT research; methods published after mid-2023 are not covered.
• Practical deployment factors (hardware specifics, serving infrastructure) are discussed qualitatively rather than with rigorous benchmarks.