As the parameters of large language models (LLMs)  expand, the computational cost of fine-tuning the entire model becomes prohibitive. To address this challenge, we introduce a parameter-efficient fine-tuning (PEFT)  method,PrincipalSingular values andSingular vectorsAdaptation (PiSSA) , which optimizes a significantly reduced parameter space while achieving or surpassing the performance of full-parameter fine-tuning. PiSSA is inspired by Intrinsic SAID, which suggests that pre-trained, over-parametrized models inhabit a space of low intrinsic dimension. Consequently, PiSSA represents a matrixwithin the model by the product of two trainable matricesand, where, plus a residual matrixfor error correction. Singular value decomposition (SVD)  is employed to factorize, and the principal singular values and vectors ofare utilized to initializeand. The residual singular values and vectors initialize the residual matrix, which keeps frozen during fine-tuning. Notably, PiSSA shares the same architecture with Low-Rank Adaptation (LoRA) , which hypothesizes that changes in model parametersform a low-rank matrix. However, LoRA approximatesthrough the product of two matrices,, initialized with Gaussian noise, and, initialized with zeros, while PiSSA initializesandwith principal singular values and singular vectors of the original matrix. Given that the principal singular values and vectors capture the essence of a low-rank matrix, PiSSA can better approximate the outcomes of full-parameter fine-tuning at the beginning by changing the essential parts while freezing the “noisy” parts. In comparison, LoRA freezes the original matrix and updates the “noise”. This distinction enables PiSSA to convergence much faster than LoRA and also achieve better performance in the end. On five common benchmarks, PiSSA outperforms LoRA on all of them using exactly the same setups except for a different initialization. On GSM8K, Mistral-7B fine-tuned with PiSSA achieves an accuracy of 72.86%, outperforming LoRA’s 67.7% by 5.16%.
Due to the same architecture, PiSSA inherits many of LoRA’s advantages, such as parameter efficiency and compatibility with quantization.
Furthermore, PiSSA reduces the 4-bit quantization error in LLaMA 2-7B by 18.97%, resulting in a substantial improvement in fine-tuning performance. On the GSM8K benchmark, PiSSA achieves an accuracy of 49.13%, surpassing the performances of QLoRA at 39.8% and LoftQ at 40.71%.
Leveraging a fast SVD technique, the initialization of PiSSA takes only a few seconds, inducing negligible cost of switching LoRA to PiSSA.