Text Generation
Transformers
PyTorch
Safetensors
code
Eval Results
Inference Endpoints
rootacess commited on
Commit
3140b3e
1 Parent(s): b9c92c5

Added Table 2

Browse files
Files changed (1) hide show
  1. README.md +323 -0
README.md CHANGED
@@ -44,6 +44,329 @@ model-index:
44
  # OctoCoder
45
 
46
  Play with the model on the [TODO Playground](https://huggingface.co/spaces/bigcode/bigcode-playground).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
47
 
48
  ## Table of Contents
49
 
 
44
  # OctoCoder
45
 
46
  Play with the model on the [TODO Playground](https://huggingface.co/spaces/bigcode/bigcode-playground).
47
+ <style>
48
+ table{
49
+ border-collapse: collapse;
50
+ }
51
+ </style>
52
+ <table>
53
+ <tr>
54
+ <th>Model (&#8595;)</th>
55
+ <th>Python</th>
56
+ <th>JavaScript</th>
57
+ <th>Java</th>
58
+ <th>Go</th>
59
+ <th>C++</th>
60
+ <th>Rust</th>
61
+ <th>Avg.</th>
62
+ </tr>
63
+ </table>
64
+ <hr style="background-color: black;">
65
+ <center><strong>HumanEvalFix</strong></center>
66
+ <hr style="background-color: black;">
67
+ <center>Non-permissive models</center>
68
+ <hr style="background-color: black;">
69
+ <table>
70
+ <tr>
71
+ <td>WizardCoder</td>
72
+ <td>31.8</td>
73
+ <td>29.5</td>
74
+ <td>12.7</td>
75
+ <td>30.4</td>
76
+ <td>18.7</td>
77
+ <td>13.0</td>
78
+ <td>22.7</td>
79
+ </tr>
80
+ <tr>
81
+ <td>GPT-4</td>
82
+ <td>47.0 </td>
83
+ <td>48.2</td>
84
+ <td>50.0</td>
85
+ <td>50.6</td>
86
+ <td>47.6</td>
87
+ <td>43.3</td>
88
+ <td><u>47.8</u></td>
89
+ </tr>
90
+ </table>
91
+ <hr style="background-color: black;">
92
+ <center>Permissive models</center>
93
+ <hr style="background-color: black;">
94
+ <table>
95
+ <tr>
96
+ <td>InstructCodeT5+<sup>‡</sup></td>
97
+ <td>2.7</td>
98
+ <td>1.2</td>
99
+ <td>4.3</td>
100
+ <td>2.1</td>
101
+ <td>0.2</td>
102
+ <td>0.5</td>
103
+ <td>1.8</td>
104
+ </tr>
105
+ <tr>
106
+ <td>BLOOMZ<sup>+</sup></td>
107
+ <td>16.6</td>
108
+ <td>15.5</td>
109
+ <td>15.2</td>
110
+ <td>16.4</td>
111
+ <td>6.7</td>
112
+ <td>5.7</td>
113
+ <td>12.5</td>
114
+ </tr>
115
+ <tr>
116
+ <td>StarChat-&#946;</td>
117
+ <td>18.1</td>
118
+ <td>18.1</td>
119
+ <td>24.1</td>
120
+ <td>18.1</td>
121
+ <td>8.2</td>
122
+ <td>3.6</td>
123
+ <td>11.2</td>
124
+ </tr>
125
+ <tr>
126
+ <td>CodeGeeX2<sup>*</sup></td>
127
+ <td>15.9</td>
128
+ <td>14.7</td>
129
+ <td>18.0</td>
130
+ <td>13.6</td>
131
+ <td>4.3</td>
132
+ <td>6.1</td>
133
+ <td>12.1</td>
134
+ </tr>
135
+ <tr>
136
+ <td>StarCoder</td>
137
+ <td>8.7</td>
138
+ <td>15.7</td>
139
+ <td>13.3</td>
140
+ <td>20.1</td>
141
+ <td>15.6</td>
142
+ <td>6.7</td>
143
+ <td>13.4</td>
144
+ </tr>
145
+ <tr>
146
+ <td>OctoGeeX<sup>*</sup></td>
147
+ <td>28.1</td>
148
+ <td>27.7</td>
149
+ <td>30.4</td>
150
+ <td>27.6</td>
151
+ <td>22.9</td>
152
+ <td>9.6</td>
153
+ <td>24.4</td>
154
+ </tr>
155
+ <tr>
156
+ <td>OctoCoder</td>
157
+ <td><strong>30.2</strong></td>
158
+ <td><strong>28.4</strong></td>
159
+ <td><strong>30.6</strong></td>
160
+ <td><strong>30.2</strong></td>
161
+ <td><strong>26.1</strong></td>
162
+ <td><strong>16.5</strong></td>
163
+ <td><strong>27.0</strong></td>
164
+ </tr>
165
+ </table>
166
+ <hr style="background-color: black;">
167
+ <center><h4>HumanEvalExplain</h4></center>
168
+ <hr style="background-color: black;">
169
+ <center>Non-permissive models</center>
170
+ <hr style="background-color: black;">
171
+ <table>
172
+ <tr>
173
+ <td>WizardCoder</td>
174
+ <td>32.5</td>
175
+ <td>33.0</td>
176
+ <td>27.4</td>
177
+ <td>26.7</td>
178
+ <td>28.2</td>
179
+ <td>16.9</td>
180
+ <td>27.5</td>
181
+ </tr>
182
+ <tr>
183
+ <td>GPT-4</td>
184
+ <td>64.6</td>
185
+ <td>57.3</td>
186
+ <td>51.2</td>
187
+ <td>58.5</td>
188
+ <td>38.4</td>
189
+ <td>42.7</td>
190
+ <td><u>52.1</u></td>
191
+ </tr>
192
+ </table>
193
+ <hr style="background-color: black;">
194
+ <center>Permissive models</center>
195
+ <hr style="background-color: black;">
196
+ <table>
197
+ <tr>
198
+ <td>InstructCodeT5+<sup>‡</sup></td>
199
+ <td>20.8</td>
200
+ <td>0.0</td>
201
+ <td>0.0</td>
202
+ <td>0.0</td>
203
+ <td>0.1</td>
204
+ <td>0.0</td>
205
+ <td>3.5</td>
206
+ </tr>
207
+ <tr>
208
+ <td>BLOOMZ<sup>+</sup></td>
209
+ <td>14.7</td>
210
+ <td>8.8</td>
211
+ <td>12.1</td>
212
+ <td>8.5</td>
213
+ <td>0.6</td>
214
+ <td>0.0</td>
215
+ <td>7.5</td>
216
+ </tr>
217
+ <tr>
218
+ <td>StarChat-&#946;</td>
219
+ <td>25.4</td>
220
+ <td>21.5</td>
221
+ <td>24.5</td>
222
+ <td>18.4</td>
223
+ <td>17.6</td>
224
+ <td>13.2</td>
225
+ <td>20.1</td>
226
+ </tr>
227
+ <tr>
228
+ <td>CodeGeeX2<sup>*</sup></td>
229
+ <td>0.0</td>
230
+ <td>0.0</td>
231
+ <td>0.0</td>
232
+ <td>0.0</td>
233
+ <td>0.0</td>
234
+ <td>0.0</td>
235
+ <td>0.0</td>
236
+ </tr>
237
+ <tr>
238
+ <td>StarCoder</td>
239
+ <td>0.0</td>
240
+ <td>0.0</td>
241
+ <td>0.0</td>
242
+ <td>0.0</td>
243
+ <td>0.0</td>
244
+ <td>0.0</td>
245
+ <td>0.0</td>
246
+ </tr>
247
+ <tr>
248
+ <td>OctoGeeX<sup>*</sup></td>
249
+ <td>30.4</td>
250
+ <td>24.0</td>
251
+ <td>24.7</td>
252
+ <td><strong>21.7</strong></td>
253
+ <td>21.0</td>
254
+ <td><strong>15.9</strong></td>
255
+ <td>22.9</td>
256
+ </tr>
257
+ <tr>
258
+ <td>OctoCoder</td>
259
+ <td><strong>35.1</strong></td>
260
+ <td><strong>24.5</strong></td>
261
+ <td><strong>27.3</strong></td>
262
+ <td>21.1</td>
263
+ <td><strong>24.1</strong></td>
264
+ <td>14.8</td>
265
+ <td><strong>24.5</strong></td>
266
+ </tr>
267
+ </table>
268
+ <hr style="background-color: black;">
269
+ <center><h4>HumanEvalSynthesize</h4></center>
270
+ <hr style="background-color: black;">
271
+ <center>Non-permissive models</center>
272
+ <hr style="background-color: black;">
273
+ <table>
274
+ <tr>
275
+ <td>WizardCoder</td>
276
+ <td>57.3</td>
277
+ <td>49.5</td>
278
+ <td>36.1</td>
279
+ <td>36.4</td>
280
+ <td>40.9</td>
281
+ <td>20.2</td>
282
+ <td>40.1</td>
283
+ </tr>
284
+ <tr>
285
+ <td>GPT-4</td>
286
+ <td>86.6</td>
287
+ <td>82.9</td>
288
+ <td>81.7</td>
289
+ <td>72.6</td>
290
+ <td>78.7</td>
291
+ <td>67.1</td>
292
+ <td><u>78.3</u></td>
293
+ </tr>
294
+ </table>
295
+ <hr style="background-color: black;">
296
+ <center>Permissive models</center>
297
+ <hr style="background-color: black;">
298
+ <table>
299
+ <tr>
300
+ <td>InstructCodeT5+<sup>‡</sup></td>
301
+ <td>37.0</td>
302
+ <td>18.9</td>
303
+ <td>17.4</td>
304
+ <td>9.5</td>
305
+ <td>19.8</td>
306
+ <td>0.3</td>
307
+ <td>17.1</td>
308
+ </tr>
309
+ <tr>
310
+ <td>BLOOMZ<sup>+</sup></td>
311
+ <td>15.6</td>
312
+ <td>14.8</td>
313
+ <td>18.4</td>
314
+ <td>8.4</td>
315
+ <td>6.5</td>
316
+ <td>5.5</td>
317
+ <td>11.5</td>
318
+ </tr>
319
+ <tr>
320
+ <td>StarChat-&#946;</td>
321
+ <td>33.5</td>
322
+ <td>31.4</td>
323
+ <td>26.7</td>
324
+ <td>25.5</td>
325
+ <td>26.6</td>
326
+ <td>14.0</td>
327
+ <td>26.3</td>
328
+ </tr>
329
+ <tr>
330
+ <td>CodeGeeX2<sup>*</sup></td>
331
+ <td>35.9</td>
332
+ <td>32.2</td>
333
+ <td>30.8</td>
334
+ <td>22.5</td>
335
+ <td>29.3</td>
336
+ <td>18.1</td>
337
+ <td>28.1</td>
338
+ </tr>
339
+ <tr>
340
+ <td>StarCoder</td>
341
+ <td>33.6</td>
342
+ <td>30.8</td>
343
+ <td>30.2</td>
344
+ <td>17.6</td>
345
+ <td>31.6</td>
346
+ <td>21.8</td>
347
+ <td>27.6</td>
348
+ </tr>
349
+ <tr>
350
+ <td>OctoGeeX<sup>*</sup></td>
351
+ <td>44.7</td>
352
+ <td>33.8</td>
353
+ <td>36.9</td>
354
+ <td>21.9</td>
355
+ <td>32.3</td>
356
+ <td>15.7</td>
357
+ <td>30.9</td>
358
+ </tr>
359
+ <tr>
360
+ <td>OctoCoder</td>
361
+ <td><strong>46.2</strong></td>
362
+ <td><strong>39.2</strong></td>
363
+ <td><strong>38.2</strong></td>
364
+ <td><strong>30.4</strong></td>
365
+ <td><strong>35.6</strong></td>
366
+ <td><strong>23.4</strong></td>
367
+ <td><strong>35.5</strong></td>
368
+ </tr>
369
+ </table>
370
 
371
  ## Table of Contents
372