Specifically, for a pre-trained weight matrix \(W_{0}\in R^{d\times k}\), LoRA constrains its update by representing the latter with a low-rank decomposition \(W_{0}+\Delta W=\hat{W}_{0}+BA\), where \(B\in R^{d\times r}\,,\,A\in R^{r\times k}\), and the rank \(r\ll min(d,k)\). During training, \(W_{0}\) is frozen and does not receive gradient updates, while \(A\) and \(B\) contain trainable parameters. It is worth mentioning that both \(W_{0}\) and \(\Delta W=BA\) are multiplied with the same input, and their respective output vectors are summed coordinate-wise. For \(h=W_{0}x\), their modified forward pass yields: \(h=W_{0}x+\Delta Wx=W_{0}x+BAx\). Usually a random Gaussian initialization is used for \(A\), and zero initialization for \(B\), so \(\Delta W=BA\) is zero at the beginning of training. They then scale \(\Delta Wx\) by \(\alpha r\), where \(\alpha\) is a constant in r.