We use the AdamW optimizer (Loshchilov and Hutter, 2019) with the hyperparameters shown in Table 4. For all model sizes, we warm up the learning rate over 5000 steps (\(\sim\) 21B tokens) and then decay it linearly from there down to a tenth of the peak learning rate over the remainder of training. After the warm-up period, we clip gradients such that the total \(l^{2}\)-norm of the parameter gradients3 does not exceed \(1.0\). Table 5 gives a comparison of our optimizer settings at the 7B scale to those of other recent LMs that also used AdamW.