\(\bullet\)_Mathematics_. Liemma [239] is proposed to enhance the mathematical capacities of general-purpose LLMs. It is developed based on CodeLLaMA. Although CodeLLaMA [235] mainly focuses on the coding ability, experiments have shown that it performs better than its base model LLaMA 2 on mathematics benchmarks [239]. Based on CodeLLaMA, Lemma is continually trained on mixtures of scientific papers, web data containing mathematical text and code (2T general tokens \(\rightarrow\) 500B code-heavy tokens \(\rightarrow\) 50\(\sim\)200B math-heavy tokens). Note that the pre-training data of Lemma also contains 5% general domain data as a form of regularization.