the learning rate using a cosine learning rate schedule set to reach its minimal value (\(\eta_{min}=0.1\cdot\eta_{max}\)) at 300B and 200B tokens, respectively. We consider warming up the learning rate for 0.5%, 1%, and 2% of \(\mathcal{D}_{1}\)'s total training iterations (132366 and 86000 iterations, respectively). Since the decay happens over the remaining budget of iterations (so resp. \(99.5\%,99\%\) and \(98\%\) of the total iterations), note that this implies that the decay phase of longer warmups happens marginally faster. Additionally, we train a model with no linear warm-up (0%) that immediately decays the LR from \(\eta_{max}\). All experiments are conducted on a 405M parameter model.