Figure 8: **Infinite learning rate schedules v.s. Cosine decay. We train a 405M parameter model on 300B tokens of SlimPajama from random initialization with two new schedules, _Cosine Inf_ and _InvSgrt Inf_, and compare them to the cosine decay baseline. _Cosine Inf_ and _InvSgrt Inf_ first decay to a fixed constant LR value and stay constant thereafter until an abrupt final decay. These schedules, therefore, have the advantage that they can smoothly transition between one pre-training phase and the next without re-warming.