The Universal Weight Subspace Hypothesis

Contributions

Universal Weight Subspace Hypothesis: Deep networks trained across diverse tasks converge to shared low-dimensional weight subspaces — a “universal subspace” that is task-agnostic and architecture-aware.
Large-Scale Empirical Validation: Analyzed 1100+ models: ~500 Mistral-7B LoRA fine-tunes, ~500 Vision Transformers (ViTs), ~50 LLaMA-8B variants across diverse domains and tasks.
Mode-Wise Spectral Analysis: Develops a tensor-mode spectral decomposition method to identify sparse joint subspaces shared across independently trained models.
Practical Implications: Demonstrates applications in model compression, efficient adaptation (LoRA rank selection), model merging, and reduced fine-tuning compute costs.

Spectral Analysis: Weight matrices W ∈ ℝ^{m×n} from independently trained models are stacked and analyzed via SVD/PCA in each tensor mode. Shared singular vectors across models identify the universal subspace.
LoRA Connection: For LoRA fine-tunes, the low-rank update matrices ΔW = BA are shown to align with the universal subspace, explaining why low-rank adaptation works so consistently.
Cross-Task Comparison: Models fine-tuned on completely different tasks (e.g., code generation vs. medical QA) share significant overlap in their weight update subspaces.
Subspace Dimensionality: The effective intrinsic dimension of the universal subspace is far smaller than the nominal parameter count — often < 1% of full parameter space.

500 Mistral-7B LoRA models across diverse tasks share a consistent low-dimensional subspace in attention weight updates
500 ViT models show similar universal subspace convergence in patch embedding and attention layers
Subspace overlap is higher in earlier layers and attention modules; later MLP layers show more task-specific variation
Model merging via subspace projection outperforms naive weight averaging (model soup)
Optimal LoRA rank can be predicted from the universal subspace dimensionality without full fine-tuning

Cross-architectural comparison (e.g., Transformer vs. CNN) left open; results may not transfer
Universal subspace may encode shared biases or failure modes, not just efficient representations
Analysis limited to supervised fine-tuning; RLHF/DPO weight subspaces not studied
Causal direction unclear: convergence to shared subspace may reflect dataset similarity rather than architecture invariant