In particular, after re-warming, models fail to recover quickly from the performance hit due to rewarming the learning rate even when training on the same dataset. This motivates finding alternatives to learning rate schedules requiring re-warming in order to improve the efficiency of continual pre-training.