We freeze the pretrained weights in the Transformer layers, and we train each rank-constrained head (i.e. in the form \(W=AB\) with \(r\) as the inner dimension of the matrix product) for various values of \(r\) on 150M tokens sampled from The Pile using 4 V100 GPUs for the Pythia models and 4 A100 GPUs for Llama-7B. We use the hyperparameters from Biderman et al. (2023), except for the batch size which we set to 256 as it fits our hardware setup better. As the trainable parameter count evolves with \(r\), we search for the best-performing learning rates among values ranging from \(1\cdot 10^{-3}\) to \(5\cdot 10^{-2}\).