Hoffmann et al. (2022) and Rae et al. (2021) established best practices for using a cosine schedule when pre-training LLMs. Specifically, they recommend starting with a linear warmup phase and decaying the learning rate to \(10\times\) its maximum value such that the end of the cosine cycle is set to match the number of tokens. While the linear warmup duration differs, most notable works have a duration between 0.1% and 0.5% for training steps (Zhao et al., 2023). Given that many popular open-source models (Touvron et al., 2023; 20; Almazrouei et al., 2023) follow this learning rate schedule recipe, it is critical to understand its nuances for continually pre-training such models.