Figure 1: We continual pretrain 1B and 7B LMs with 15B OpenWebMath tokens. Rho-1 is trained with our proposed Selective Language Modeling (SLM), while baselines are trained using causal language modeling. SLM improves average few-shot accuracy on GSM8k and MATH by over 16%, achieving the baseline performance 5-10x faster.