--- pipeline_tag: text-generation license: other language: - en - zh tags: - math base_model: internlm/internlm2-math-plus-7b --- # InternLM-Math-Plus-GGUF This is quantized version of [internlm/internlm2-math-plus-7b](https://huggingface.co/internlm/internlm2-math-plus-7b) created using llama.cpp # Model Description ## News - [2024.05.24] We release updated version InternLM2-Math-Plus with 4 sizes and state-of-the-art performances including 1.8B, 7B, 20B, and 8x22B. We improve informal math reasoning performance (chain-of-thought and code-intepreter) and formal math reasoning performance (LEAN 4 translation and LEAN 4 theorem proving) significantly. - [2024.02.10] We add tech reports and citation reference. - [2024.01.31] We add MiniF2F results with evaluation codes! - [2024.01.29] We add checkpoints from ModelScope. Update results about majority voting and Code Intepreter. Tech report is on the way! - [2024.01.26] We add checkpoints from OpenXLab, which ease Chinese users to download! ## Performance ## Formal Math Reasoning We evaluate the performance of InternLM2-Math-Plus on formal math reasoning benchmark MiniF2F-test. The evaluation setting is same as Llemma with LEAN 4. | Models | MiniF2F-test | | -------------------------------- | ------------ | | ReProver | 26.5 | | LLMStep | 27.9 | | GPT-F | 36.6 | | HTPS | 41.0 | | Llemma-7B | 26.2 | | Llemma-34B | 25.8 | | InternLM2-Math-7B-Base | 30.3 | | InternLM2-Math-20B-Base | 29.5 | | InternLM2-Math-Plus-1.8B | 38.9 | | InternLM2-Math-Plus-7B | **43.4** | | InternLM2-Math-Plus-20B | 42.6 | | InternLM2-Math-Plus-Mixtral8x22B | 37.3 | ## Informal Math Reasoning We evaluate the performance of InternLM2-Math-Plus on informal math reasoning benchmark MATH and GSM8K. InternLM2-Math-Plus-1.8B outperforms MiniCPM-2B in the smallest size setting. InternLM2-Math-Plus-7B outperforms Deepseek-Math-7B-RL which is the state-of-the-art math reasoning open source model. InternLM2-Math-Plus-Mixtral8x22B achieves 68.5 on MATH (with Python) and 91.8 on GSM8K. | Model | MATH | MATH-Python | GSM8K | | -------------------------------- | -------- | ----------- | -------- | | MiniCPM-2B | 10.2 | - | 53.8 | | InternLM2-Math-Plus-1.8B | **37.0** | **41.5** | **58.8** | | InternLM2-Math-7B | 34.6 | 50.9 | 78.1 | | Deepseek-Math-7B-RL | 51.7 | 58.8 | **88.2** | | InternLM2-Math-Plus-7B | **53.0** | **59.7** | 85.8 | | InternLM2-Math-20B | 37.7 | 54.3 | 82.6 | | InternLM2-Math-Plus-20B | **53.8** | **61.8** | **87.7** | | Mixtral8x22B-Instruct-v0.1 | 41.8 | - | 78.6 | | Eurux-8x22B-NCA | 49.0 | - | - | | InternLM2-Math-Plus-Mixtral8x22B | **58.1** | **68.5** | **91.8** | We also evaluate models on [MathBench-A](https://github.com/open-compass/MathBench). InternLM2-Math-Plus-Mixtral8x22B has comparable performance compared to Claude 3 Opus. | Model | Arithmetic | Primary | Middle | High | College | Average | | -------------------------------- | ---------- | ------- | ------ | ---- | ------- | ------- | | GPT-4o-0513 | 77.7 | 87.7 | 76.3 | 59.0 | 54.0 | 70.9 | | Claude 3 Opus | 85.7 | 85.0 | 58.0 | 42.7 | 43.7 | 63.0 | | Qwen-Max-0428 | 72.3 | 86.3 | 65.0 | 45.0 | 27.3 | 59.2 | | Qwen-1.5-110B | 70.3 | 82.3 | 64.0 | 47.3 | 28.0 | 58.4 | | Deepseek-V2 | 82.7 | 89.3 | 59.0 | 39.3 | 29.3 | 59.9 | | Llama-3-70B-Instruct | 70.3 | 86.0 | 53.0 | 38.7 | 34.7 | 56.5 | | InternLM2-Math-Plus-Mixtral8x22B | 77.5 | 82.0 | 63.6 | 50.3 | 36.8 | 62.0 | | InternLM2-Math-20B | 58.7 | 70.0 | 43.7 | 24.7 | 12.7 | 42.0 | | InternLM2-Math-Plus-20B | 65.8 | 79.7 | 59.5 | 47.6 | 24.8 | 55.5 | | Llama3-8B-Instruct | 54.7 | 71.0 | 25.0 | 19.0 | 14.0 | 36.7 | | InternLM2-Math-7B | 53.7 | 67.0 | 41.3 | 18.3 | 8.0 | 37.7 | | Deepseek-Math-7B-RL | 68.0 | 83.3 | 44.3 | 33.0 | 23.0 | 50.3 | | InternLM2-Math-Plus-7B | 61.4 | 78.3 | 52.5 | 40.5 | 21.7 | 50.9 | | MiniCPM-2B | 49.3 | 51.7 | 18.0 | 8.7 | 3.7 | 26.3 | | InternLM2-Math-Plus-1.8B | 43.0 | 43.3 | 25.4 | 18.9 | 4.7 | 27.1 |