We presentLlemma, a large language model for mathematics.
We continue pretraining Code Llama on--, a mixture of scientific papers, web data containing mathematics, and mathematical code, yieldingLlemma.
On the MATH benchmarkLlemmaoutperforms all known open base models, as well as the unreleased Minerva model suite on an equi-parameter basis. Moreover,Llemmais capable of tool use and formal theorem proving without any further finetuning.
We openly release all artifacts, including 7 billion and 34 billion parameter models, the--, and code to replicate our experiments.111https://github.com/EleutherAI/math-lm