Update README.md
Browse files
README.md
CHANGED
@@ -65,28 +65,30 @@ The model was trained according to the OLM GPT2 instructions at this [repo](http
|
|
65 |
|
66 |
The model achieves the following results without any fine-tuning (zero-shot):
|
67 |
|
68 |
-
| Task |
|
69 |
-
|
70 |
-
|rte |
|
71 |
-
|piqa |
|
72 |
-
|copa |
|
73 |
-
|record |
|
74 |
-
|boolq |
|
75 |
-
|cb |
|
76 |
-
|hellaswag |
|
77 |
-
|mrpc |
|
78 |
-
|multirc |
|
79 |
-
|lambada |
|
80 |
-
|wsc |
|
81 |
-
|wic |
|
82 |
-
|mnli |
|
83 |
-
|qnli |
|
84 |
-
|cola |
|
85 |
-
|triviaqa |
|
86 |
-
|winogrande |
|
87 |
-
|webqs |
|
88 |
-
|arc_easy |
|
89 |
-
|arc_challenge|
|
90 |
-
|
91 |
-
To get these results, we used the Eleuther AI evaluation harness [here](https://github.com/EleutherAI/lm-evaluation-harness),
|
92 |
-
which can produce results different than those reported in the GPT2 paper.
|
|
|
|
|
|
65 |
|
66 |
The model achieves the following results without any fine-tuning (zero-shot):
|
67 |
|
68 |
+
| Task | Metric | Original GPT2 | OLM GPT2 Dec 2022 (Ours) | Significance of Difference (two-tailed p-value) |
|
69 |
+
|:------------|:-----------|--------------------:|-------------------------:|----------------------------------:|
|
70 |
+
|rte |acc |0.5307 |0.5199 |0.7184 |
|
71 |
+
|piqa |acc/acc_norm|0.6289/0.6251 |**0.6692**/**0.6665** |**0.0004**/**0.0003** |
|
72 |
+
|copa |acc |0.6400 |0.6800 |0.4070 |
|
73 |
+
|record |f1/em |**0.7094**/**0.7026**|0.6884/0.6818 |**0.0000**/**0.0000** |
|
74 |
+
|boolq |acc |0.4872 |**0.6021** |**0.0000** |
|
75 |
+
|cb |acc/f1 |0.4107/0.2619 |0.3393/0.1840 |0.2816/NA |
|
76 |
+
|hellaswag |acc/acc_norm|0.2892/0.3114 |**0.3079**/**0.3482** |**0.0000**/**0.0000** |
|
77 |
+
|mrpc |acc/f1 |0.5662/0.6911 |**0.6814**/**0.8099** |**0.0000**/**0.0000** |
|
78 |
+
|multirc |acc |0.0189 |0.0220 |0.4755 |
|
79 |
+
|lambada |ppl/acc |40.0554/0.3256 |**28.3359**/**0.3699** |**0.0000**/**0.0000** |
|
80 |
+
|wsc |acc |0.4327 |0.3654 |0.1680 |
|
81 |
+
|wic |acc |0.4922 |0.5000 |0.6924 |
|
82 |
+
|mnli |acc |0.3372 |**0.3501** |**0.0071** |
|
83 |
+
|qnli |acc |0.5017 |0.4946 |0.2913 |
|
84 |
+
|cola |mcc |0.0126 |0.0000 |0.6880 |
|
85 |
+
|triviaqa |acc |0.0151 |**0.0181** |**0.0088** |
|
86 |
+
|winogrande |acc |0.5162 |0.5051 |0.4314 |
|
87 |
+
|webqs |acc |0.0030 |**0.0079** |**0.0000** |
|
88 |
+
|arc_easy |acc/acc_norm|0.4381/0.3948 |**0.4693**/**0.4230** |**0.0022**/**0.0049** |
|
89 |
+
|arc_challenge|acc/acc_norm|0.1903/0.2270 |0.2090/0.2398 |0.1017/0.2957 |
|
90 |
+
|
91 |
+
To get these results, we used commit `f079e322b857714fcef1ada9e78ddc606fe51e84` of the Eleuther AI evaluation harness [here](https://github.com/EleutherAI/lm-evaluation-harness),
|
92 |
+
which can produce results different than those reported in the GPT2 paper.
|
93 |
+
We added a change [here](https://github.com/EleutherAI/lm-evaluation-harness/compare/master...mathemakitten:lm-evaluation-harness:master) to enable evaluation of the OLM GPT2, which has a very slightly different vocab size.
|
94 |
+
The p-values come from the stderr from the evaluation harness, plus a normal distribution assumption.
|