zhangir-azerbayev commited on
Commit
f99619d
1 Parent(s): b2ce480

update readme

Browse files
Files changed (3) hide show
  1. README.md +18 -8
  2. llemma.jpg +0 -0
  3. llemma.png +0 -0
README.md CHANGED
@@ -8,7 +8,7 @@ tags:
8
  - math
9
  - reasoning
10
  ---
11
- <img src="llemma.jpg" width="400">
12
 
13
  [Zhangir Azerbayev](https://zhangir-azerbayev.github.io/), [Hailey Schoelkopf](https://github.com/haileyschoelkopf), [Keiran Paster](https://keirp.com), [Marco Dos Santos](https://github.com/dsantosmarco), [Stephen McAleer](https://www.andrew.cmu.edu/user/smcaleer/), [Albert Q. Jiang](https://albertqjiang.github.io/), [Jia Deng](https://www.cs.princeton.edu/~jiadeng/), [Stella Biderman](https://www.stellabiderman.com/), [Sean Welleck](https://wellecks.com/)
14
 
@@ -29,12 +29,12 @@ On chain-of-thought mathematics tasks, Llemma models outperform Llama-2, Code Ll
29
  | Model | Size | GSM8k | [OCW](https://openreview.net/forum?id=IFXTZERXdM7) | MMLU-STEM | [SAT](https://huggingface.co/datasets/mcaleste/sat_multiple_choice_math_may_23) | MATH |
30
  |------------|------|--------|-------|-----------|-------|-------|
31
  | Llama 2 | 7B | 11.8% | 3.7% | 29.9% | 25% | 3.2% |
32
- | Code Llama | 7B | 10.5% | 4.4% | 25.1% | 9.4% | 4.4% |
33
- | LLEMMA | 7B | 36.4% | 7.7% | 37.7% | 53.1% | 17.2% |
34
- | Minerva | 8B | 16.2% | 7.7% | 35.6% | - | 14.1% |
35
  |------------|------|--------|-------|-----------|-------|-------|
36
- | Code Llama | 34B | 29.6% | 7.0% | 40.5% | 40.6% | 11.9% |
37
- | LLEMMA | 34B | 51.5% | 11.8% | 49.0% | 71.9% | 24.1% |
38
  |------------|------|--------|-------|-----------|-------|-------|
39
  | Minerva | 62B | 52.4% | 12.0% | 53.9% | - | 27.6% |
40
  | Minerva | 540B | 58.8% | 17.6% | 63.9% | - | 33.6% |
@@ -44,10 +44,10 @@ Further performance can be extracted by using majority voting:
44
 
45
  | Model | Size | GSM8k maj@100 | OCW maj@100 | MMLU-STEM maj@16 | SAT maj@16 | MATH maj@256 |
46
  |---------|------|-------------|-----------|-----------------|-----------|------------|
47
- | LLEMMA | 7B | 54.0% | 14.3% | 49.9% | 78.1% | 32.0% |
48
  | Minerva | 8B | 28.4% | 12.5% | 43.4% | - | 25.4% |
49
  |---------|------|-------------|-----------|-----------------|-----------|------------|
50
- | LLEMMA | 34B | 69.3% | 18.4% | 59.7% | 81.3% | 41.0% |
51
  |---------|------|-------------|-----------|-----------------|-----------|------------|
52
  | Minerva | 62B | 68.5% | 23.5% | 63.5% | - | 43.4% |
53
  | Minerva | 540B | 78.5% | 30.8% | 75.0% | - | 50.3% |
@@ -55,5 +55,15 @@ Further performance can be extracted by using majority voting:
55
  ### Tool Use and Theorem Proving
56
  In addition to chain-of-thought reasoning, Llemma has strong capabilities in computational mathematics tasks. For tool use and formal theorem proving evaluations, see [our paper](#).
57
 
 
 
 
 
 
 
 
 
 
 
58
 
59
 
 
8
  - math
9
  - reasoning
10
  ---
11
+ <img src="llemma.png" width="400">
12
 
13
  [Zhangir Azerbayev](https://zhangir-azerbayev.github.io/), [Hailey Schoelkopf](https://github.com/haileyschoelkopf), [Keiran Paster](https://keirp.com), [Marco Dos Santos](https://github.com/dsantosmarco), [Stephen McAleer](https://www.andrew.cmu.edu/user/smcaleer/), [Albert Q. Jiang](https://albertqjiang.github.io/), [Jia Deng](https://www.cs.princeton.edu/~jiadeng/), [Stella Biderman](https://www.stellabiderman.com/), [Sean Welleck](https://wellecks.com/)
14
 
 
29
  | Model | Size | GSM8k | [OCW](https://openreview.net/forum?id=IFXTZERXdM7) | MMLU-STEM | [SAT](https://huggingface.co/datasets/mcaleste/sat_multiple_choice_math_may_23) | MATH |
30
  |------------|------|--------|-------|-----------|-------|-------|
31
  | Llama 2 | 7B | 11.8% | 3.7% | 29.9% | 25% | 3.2% |
32
+ | Code Llama | 7B | 10.5% | 4.4% | 25.1% | 9.4% | 4.5% |
33
+ | LLEMMA | 7B | **36.4%** | **7.7%** | **37.7%** | **53.1%** | **18.0%** |
34
+ | Minerva | 8B | 16.2% | **7.7%** | 35.6% | - | 14.1% |
35
  |------------|------|--------|-------|-----------|-------|-------|
36
+ | Code Llama | 34B | 29.6% | 7.0% | 40.5% | 40.6% | 12.2% |
37
+ | LLEMMA | 34B | **51.5%** | **11.8%** | **49.0%** | **71.9%** | **25.0%** |
38
  |------------|------|--------|-------|-----------|-------|-------|
39
  | Minerva | 62B | 52.4% | 12.0% | 53.9% | - | 27.6% |
40
  | Minerva | 540B | 58.8% | 17.6% | 63.9% | - | 33.6% |
 
44
 
45
  | Model | Size | GSM8k maj@100 | OCW maj@100 | MMLU-STEM maj@16 | SAT maj@16 | MATH maj@256 |
46
  |---------|------|-------------|-----------|-----------------|-----------|------------|
47
+ | LLEMMA | 7B | 54.0% | 14.3% | 49.9% | 78.1% | **33.5** |
48
  | Minerva | 8B | 28.4% | 12.5% | 43.4% | - | 25.4% |
49
  |---------|------|-------------|-----------|-----------------|-----------|------------|
50
+ | LLEMMA | 34B | 69.3% | 18.4% | 59.7% | 81.3% | **43.1%** |
51
  |---------|------|-------------|-----------|-----------------|-----------|------------|
52
  | Minerva | 62B | 68.5% | 23.5% | 63.5% | - | 43.4% |
53
  | Minerva | 540B | 78.5% | 30.8% | 75.0% | - | 50.3% |
 
55
  ### Tool Use and Theorem Proving
56
  In addition to chain-of-thought reasoning, Llemma has strong capabilities in computational mathematics tasks. For tool use and formal theorem proving evaluations, see [our paper](#).
57
 
58
+ ### Citation
59
+ ```
60
+ @article{azerbayev2023llemma,
61
+ title={Llemma: an open language model for mathematics},
62
+ author={Zhangir Azerbayev and Hailey Schoelkopf and Keiran Paster and Marco Dos Santos and Stephen McAleer and Albert Q. Jiang and Jia Deng and Stella Biderman and Sean Welleck},
63
+ eprint={xyz.xyz},
64
+ archivePrefix={arXiv}
65
+ year={2023}
66
+ }
67
+ ```
68
 
69
 
llemma.jpg DELETED
Binary file (269 kB)
 
llemma.png ADDED