We now investigate the benefits of re-warming and re-decaying the learning rate (e.g., following a cosine schedule) for different values of \(\eta_{\text{max}}\). Specifically, we compare these models to two natural baselines: a model that does not re-warm, staying constant at \(\eta_{\text{min}}\) (\(3\cdot 10^{-5}\)), and a model that re-warms to the pre-training \(\eta_{\text{max}}\) (\(3\cdot 10^{-4}\)) but does not re-decay. We use the same two-dataset settings: we first pre-train on the Pile (\(\mathcal{D}_{0}\)) for 300B tokens and continually pre-train our model on SlimPjama (weak shift) or German Common Crawl (strong shift) as our \(\mathcal{D}_{1}\) datasets. The continual pre-training is conducted for the full size (300B and 200B tokens, respectively) of the datasets. The models that re-warm and re-decay the LR consider three strategies: re-warming to half the pre-training's \(\eta_{\text{max}}\) (\(1.5\cdot 10^{-4}\)), re-warming to the same \(\eta_{\text{max}}\) as pre-training (\(3\cdot 10^{-4}\)), and re-warming to twice the \(\eta_{\text{max}}\) of pre-training (\(6\cdot 10^{-4}\)). In all cases, the learning rate is cosine-decayed after linear warmup to reach \(\eta_{\text{min}}=0.1\cdot\eta_{\text{max}}\) by the end of training. Finally, we consider models trained on \(\mathcal{D}_{0}\cup\mathcal{D}_{1}\) as a third baseline to provide an upper bound on performance.