While LLMs are the cornerstone of current generative AI technology, they are prohibitively expensive to train and keep up to date. However, as new and higher-quality datasets continue to become available (Gao et al., 2020; Soboleva et al., 2023; Computer, 2023; Soldaini et al., 2024), organizations will need to update their models to stay abreast of the competition. Currently, LLMs are re-trained on a combination of old and newly collected data. Existing works aim to reduce these training costs by enabling low-cost hyperparameter optimization (Yang et al., 2022) or providing guidelines for maximizing performance under a given compute budget (Hoffmann et al., 2022). However, these works assume that models will be _trained from random initialization_, raising the following question: Should practitioners always combine existing datasets and _train from random initialization_ to obtain the best performance? Doing so for every update of the models quickly becomes prohibitively expensive.