SLM on math continual pretraining: both 1B and 7B Rho-1 outperform CLM-trained baselines by over 16% on the GSM8k and MATH datasets. SLM reaches baseline accuracy up to 10x faster, as shown in Figure 1. Remarkably, Rho-1-7B matches the state-of-the-art performance of DeepSeekMath-7B using only 15B tokens, compared to the 500B tokens required by DeepSeekMath. Upon fine-tuning, Rho-1-1B and 7B achieve 40.6% and 51.8% on MATH, respectively. Notably, Rho-1-1B is the first 1B LM to exceed 40% accuracy, nearing the early GPT-4's CoT performance of 42.5%. SS3.3 confirms the efficacy of SLM in general pretraining: Training Tinyllama-1B on 80B tokens with SLM improves 6.8% on average across 15 benchmarks, with gains over 10% in code and math tasks.