Given the influence that the learning rate can have on adaptation and the low final LR values of prominent LLMs (Rae et al., 2021; Hoffmann et al., 2022; Zhao et al., 2023; Touvron et al., 2023;a), we hypothesize that the LR should be re-warmed and re-decayed to promote adaptation during continual pre-training. In this section, we investigate the effect of linear warmup duration, re-warming the LR, re-decaying the LR, and the magnitude of the \(\eta_{max}\) when continuing to pre-train. Specifically, we evaluate their respective effects in the **two-dataset weak shift** setting (300B Pile \(\rightarrow\) 300B SlimPajama) and the **two-dataset stronger shift** setting (300B Pile \(\rightarrow\) 300B SlimPajama). Notably, the model trained on \(\mathcal{D}_{0}\) (300B tokens of Pile) follow a linear warmup and cosine decay schedule1, simulating many common open-source pre-trained LLMs.