* If the learning rate was cosine-decayed from a large value \(\eta_{max}\) to a small value \(\eta_{min}\) during pre-training on the initial dataset, the following guidelines can help to continually pre-train your model:
* Re-warming and re-decaying the learning rate from \(\mathcal{O}(\eta_{max})\) to \(\mathcal{O}(\eta_{min})\) improves adaptation to a new dataset, e.g. compared to continuing from small learning rates \(\mathcal{O}(\eta_{min})\). * Decreasing the schedule's maximum learning rate can help reduce forgetting, whereas increasing it can improve adaptation. * Infinite LR schedules are promising alternatives to cosine decay schedules. They transition into a high constant learning rate across tasks, helping prevent optimization-related forgetting by avoiding re-warming the LR between tasks. They also avoid committing to a specific budget of tokens as a final exponential decay can be used to train the model to convergence at any point during training. **Replay:**
* As a default value, we recommend 5% replay. More replay should be used with stronger distribution shifts, while one can get away with as little as 1% for weak distribution shifts.