Pre-trainingWe train all our XL models for a total of 300k steps with a batch size of one million tokens, unless otherwise specified. We use a maximum learning rate of \(3e^{-4}\) for the 128M, and 350M parameter models, and \(2e^{-4}\) for the 1.3B parameter model. The minimum learning rate is \(1e^{-5}\). We use a weight decay of 0.01, along with a gradient clipping norm of 1.0. We use cosine learning rate scheduler with a warmup for 1% of the total steps; and the Adam optimizer with \(\beta_{1}=0.9\) and \(\beta_{2}=0.999\).