* 1 Introduction
* 2 Main findings and Takeaways
* 3 Related Work
	* 3.1 Continual learning
	* 3.2 Pre-training, Model Scale, and Continual Learning
	* 3.3 Domain Adaptive Continual Pre-training (DACPT)
	* 3.4 Continual Learning for LMs Applied to Specific Domains
	* 3.5 Learning Rate Schedules
* 4 Background & Methodology
	* 4.1 Linear Warmup and Cosine Decay Schedule
	* 4.2 Compute-equivalent Replay
* 5 Experimental Setup
	* 5.1 Datasets
	* 5.2 Continual Learning Settings
	* 5.3 Training Setup
	* 5.4 German and English LM Evaluation Benchmark
* 6 Results
	* 6.1 Learning Rate Schedule
		* 6.1.1 The Effect of Linear Warmup for Weak and Strong Distribution Shifts. 		* 6.1.2 The effect of re-warming, re-decaying, and varying \(\eta_{max}\) for Weak and Strong Distribution Shifts. 	* 6.2 The Effect of Replay
	* 6.3 Continual Pre-training Final Performance for Weak and Strong Distribution Shifts.