In this paper, we show that Low Rank Adaptation (LoRA)  as originally introduced inleads to suboptimal finetuning of models with large width (embedding dimension) . This is due to the fact that adapter matricesandin LoRA are updated with the same learning rate. Using scaling arguments for large width networks, we demonstrate that using the same learning rate foranddoes not allow efficient feature learning. We then show that this suboptimality of LoRA can be corrected simply by setting different learning rates for the LoRA adapter matricesandwith a well-chosen fixed ratio. We call this proposed algorithm LoRA. In our extensive experiments, LoRAimproves performance (improvements)  and finetuning speed (up toX SpeedUp) , at the same computational cost as LoRA.