We now explore the role of the infinite learning rate schedules when multiple new datasets are seen in a continual learning setup. The models are trained from random initialization with different learning rate schedules on 3 IID 100B subsets of SlimPajama (e.g., our _three datasets no shift_ setting; see Sec 5.2). We focus on the no shift setting in these preliminary experiments and leave the weak and strong shift cases to future work. This task simulates a setting where large amounts of data from the same distribution are received at time increments and we wish to continue pre-training our models on them (e.g., continuing to pre-train the model on the latest web-scrape). To make our results applicable to situations where previous optimizer states are not available, we do not keep optimizer states across dataset boundaries. Fig. 9 reports training curves for 405M parameter models.