**Low-Rank Adaptation (LoRA)**. LoRA [145] imposes the low-rank constraint for approximating the update matrix at each dense layer, so as to reduce the trainable parameters for adapting to downstream tasks. Consider the case of optimizing a parameter matrix \(\mathbf{W}\). The update process can be written in a general form as: \(\mathbf{W}\leftarrow\mathbf{W}+\Delta\mathbf{W}\). The basic idea of LoRA is to freeze the original matrix \(\mathbf{W}\in\mathbb{R}^{m\times n}\) while approximating the parameter update \(\Delta\mathbf{W}\) by low-rank decomposition matrices, _i.e.,_\(\Delta\mathbf{W}=\mathbf{A}\cdot\mathbf{B}^{\top}\), where \(\mathbf{A}\in\mathbb{R}^{m\times k}\) and \(\mathbf{B}\in\mathbb{R}^{n\times k}\) are the trainable parameters for task adaptation and \(k\ll\min(m,n)\) is the reduced rank. The major merit of LoRA is that it can largely save the memory and storage usage (_e.g.,_ VRAM). Further, one can only keep a single large model copy, while maintaining a number of task-specific low-rank decomposition matrices for adapting to different downstream tasks. Further, several studies have also discussed how to set the rank in a more principled approach, _e.g.,_ importance score based allocation [406] and search-free optimal rank selection [407].