code region:us
---
dataset_info:
  features:
  - name: experiment
    dtype: string
  - name: problem
    dtype: string
  - name: language
    dtype: string
  - name: top_p
    dtype: float64
  - name: max_tokens
    dtype: int64
  - name: prompt
    dtype: string
  - name: tests
    dtype: string
  - name: stop_tokens
    sequence: string
  - name: completions
    sequence: string
  - name: programs
    sequence: string
  - name: stdouts
    sequence: string
  - name: stderrs
    sequence: string
  - name: exit_codes
    sequence: int64
  - name: statuses
    sequence: string
  - name: timestamps
    sequence: int64
  splits:
  - name: humaneval.py.bigcode_15b_800m.0.2.reworded
    num_bytes: 50941974
    num_examples: 161
  - name: humaneval.py.bigcode_15b_200m.0.2.reworded
    num_bytes: 57850786
    num_examples: 161
  - name: humaneval.py.bigcode_15b_400m.0.2.reworded
    num_bytes: 52404545
    num_examples: 161
  - name: humaneval.py.bigcode_15b_600m.0.2.reworded
    num_bytes: 55071293
    num_examples: 161
  - name: humaneval.rkt.bigcode_15b_800m.0.2.reworded
    num_bytes: 77194321
    num_examples: 161
  - name: humaneval.py.chatgpt.0.2.reworded
    num_bytes: 5473126
    num_examples: 161
  - name: humaneval.r.bigcode_15b_800m.0.2.reworded
    num_bytes: 73160389
    num_examples: 161
  - name: humaneval.r.bigcode_15b_1000m.0.2.reworded
    num_bytes: 63088872
    num_examples: 161
  - name: humaneval.r.bigcode_15b_200m.0.2.reworded
    num_bytes: 77532102
    num_examples: 161
  - name: humaneval.r.bigcode_15b_400m.0.2.reworded
    num_bytes: 80103073
    num_examples: 161
  - name: humaneval.r.bigcode_15b_600m.0.2.reworded
    num_bytes: 76123407
    num_examples: 161
  - name: humaneval.py.bigcode_15b_1000m.0.2.reworded
    num_bytes: 47282373
    num_examples: 161
  - name: humaneval.py.bigcode_15b_1000m.0.1.reworded
    num_bytes: 11724023
    num_examples: 161
  - name: humaneval.py.bigcode_15b_1000m.0.4.reworded
    num_bytes: 12167610
    num_examples: 161
  - name: humaneval.py.bigcode_15b_1000m.0.6.reworded
    num_bytes: 12344190
    num_examples: 161
  - name: humaneval.py.bigcode_15b_1000m.0.8.reworded
    num_bytes: 12825651
    num_examples: 161
  - name: humaneval.py.codegeex.0.2.reworded
    num_bytes: 49547494
    num_examples: 161
  - name: humaneval.py.codegen.0.2.reworded
    num_bytes: 55391634
    num_examples: 161
  - name: humaneval.py.Salesforce_codegen_16B_mono.0.2.reworded
    num_bytes: 54757013
    num_examples: 161
  - name: humaneval.py.cushman001.0.2.reworded
    num_bytes: 5081696
    num_examples: 161
  - name: humaneval.js.pystarcoder2.0.2.reworded
    num_bytes: 6784781
    num_examples: 161
  - name: humaneval.py.santacoder.0.2.reworded
    num_bytes: 57098223
    num_examples: 161
  - name: humaneval.py.santacoder_fim_prompt.0.2.reworded
    num_bytes: 5473782
    num_examples: 161
  - name: humaneval.lua.bigcode_15b_1000m.0.2.reworded
    num_bytes: 53170918
    num_examples: 161
  - name: mbpp.py.bigcode_15b_1000m.0.2.reworded
    num_bytes: 76438518
    num_examples: 397
  - name: mbpp.py.pystarcoder2.0.2.reworded
    num_bytes: 78424728
    num_examples: 397
  - name: mbpp.lua.davinci.0.2.reworded
    num_bytes: 82574073
    num_examples: 401
  - name: humaneval.js.davinci.0.2.transform
    num_bytes: 61113074
    num_examples: 161
  - name: humaneval.py.davinci.0.2.transform
    num_bytes: 46653237
    num_examples: 161
  - name: humaneval.ts.davinci.0.2.transform
    num_bytes: 60332695
    num_examples: 159
  - name: humaneval.ts.davinci.0.2.reworded
    num_bytes: 60256365
    num_examples: 159
  - name: humaneval.cpp.davinci.0.2.reworded
    num_bytes: 67044215
    num_examples: 159
  - name: humaneval.cs.davinci.0.2.reworded
    num_bytes: 103369582
    num_examples: 156
  - name: humaneval.d.davinci.0.2.reworded
    num_bytes: 66641398
    num_examples: 156
  - name: humaneval.go.davinci.0.2.reworded
    num_bytes: 71050586
    num_examples: 154
  - name: humaneval.java.davinci.0.2.reworded
    num_bytes: 71969268
    num_examples: 156
  - name: humaneval.jl.davinci.0.2.reworded
    num_bytes: 76515733
    num_examples: 159
  - name: humaneval.js.davinci.0.2.reworded
    num_bytes: 61674621
    num_examples: 161
  - name: humaneval.lua.davinci.0.2.reworded
    num_bytes: 54016568
    num_examples: 161
  - name: humaneval.php.davinci.0.2.reworded
    num_bytes: 61403511
    num_examples: 161
  - name: humaneval.pl.davinci.0.2.reworded
    num_bytes: 60281076
    num_examples: 161
  - name: humaneval.py.davinci.0.2.reworded
    num_bytes: 48223052
    num_examples: 161
  - name: humaneval.rb.davinci.0.2.reworded
    num_bytes: 65236003
    num_examples: 161
  - name: humaneval.r.davinci.0.2.reworded
    num_bytes: 48031115
    num_examples: 161
  - name: humaneval.rkt.davinci.0.2.reworded
    num_bytes: 66273931
    num_examples: 161
  - name: humaneval.rs.davinci.0.2.reworded
    num_bytes: 56522566
    num_examples: 156
  - name: humaneval.scala.davinci.0.2.reworded
    num_bytes: 84153007
    num_examples: 158
  - name: humaneval.sh.davinci.0.2.reworded
    num_bytes: 58089427
    num_examples: 158
  - name: humaneval.swift.davinci.0.2.reworded
    num_bytes: 61663115
    num_examples: 158
  - name: humaneval.java.bigcode_15b_1000m.0.2.reworded
    num_bytes: 75580922
    num_examples: 158
  - name: humaneval.java.bigcode_15b_200m.0.2.reworded
    num_bytes: 84307515
    num_examples: 158
  - name: humaneval.java.bigcode_15b_400m.0.2.reworded
    num_bytes: 85145220
    num_examples: 158
  - name: humaneval.java.bigcode_15b_600m.0.2.reworded
    num_bytes: 80223655
    num_examples: 158
  - name: humaneval.java.bigcode_15b_800m.0.2.reworded
    num_bytes: 78345454
    num_examples: 158
  - name: humaneval.java.codegeex.0.2.reworded
    num_bytes: 74859734
    num_examples: 158
  - name: humaneval.java.codegen.0.2.reworded
    num_bytes: 76705002
    num_examples: 158
  - name: humaneval.java.cushman001.0.2.reworded
    num_bytes: 7860926
    num_examples: 158
  - name: humaneval.java.replit_code.0.2.reworded
    num_bytes: 20396812
    num_examples: 158
  - name: humaneval.java.pystarcoder2.0.2.reworded
    num_bytes: 73528078
    num_examples: 158
  - name: humaneval.java.davinci.0.2.keep
    num_bytes: 71389774
    num_examples: 161
  - name: humaneval.java.davinci.0.2.remove
    num_bytes: 64493441
    num_examples: 157
  - name: humaneval.java.davinci.0.2.transform
    num_bytes: 72660740
    num_examples: 156
  - name: humaneval.java.davinci.0.8.keep
    num_bytes: 72358729
    num_examples: 161
  - name: humaneval.java.davinci.0.8.reworded
    num_bytes: 75043862
    num_examples: 156
  - name: humaneval.java.incoder.0.2.keep
    num_bytes: 110176353
    num_examples: 161
  - name: humaneval.java.incoder.0.2.remove
    num_bytes: 88903768
    num_examples: 157
  - name: humaneval.java.incoder.0.2.reworded
    num_bytes: 109021885
    num_examples: 156
  - name: humaneval.java.incoder.0.2.transform
    num_bytes: 107183302
    num_examples: 156
  - name: humaneval.java.incoder.0.8.keep
    num_bytes: 75299144
    num_examples: 139
  - name: humaneval.java.incoder.0.8.reworded
    num_bytes: 100533855
    num_examples: 158
  - name: mbpp.java.codegen.0.2.reworded
    num_bytes: 144592215
    num_examples: 373
  - name: mbpp.java.codegen.0.8.reworded
    num_bytes: 47521423
    num_examples: 120
  - name: mbpp.java.davinci.0.2.keep
    num_bytes: 135567713
    num_examples: 373
  - name: mbpp.java.davinci.0.2.reworded
    num_bytes: 136848151
    num_examples: 373
  - name: mbpp.java.incoder.0.2.reworded
    num_bytes: 225046095
    num_examples: 373
  - name: mbpp.java.incoder.0.8.reworded
    num_bytes: 167923488
    num_examples: 373
  - name: humaneval.cpp.davinci.0.2.keep
    num_bytes: 63794632
    num_examples: 161
  - name: humaneval.cpp.davinci.0.2.remove
    num_bytes: 58355394
    num_examples: 158
  - name: humaneval.cpp.davinci.0.2.transform
    num_bytes: 66852210
    num_examples: 159
  - name: humaneval.cpp.davinci.0.8.keep
    num_bytes: 61668425
    num_examples: 161
  - name: humaneval.cpp.bigcode_15b_1000m.0.2.reworded
    num_bytes: 67353068
    num_examples: 161
  - name: humaneval.cpp.bigcode_15b_200m.0.2.reworded
    num_bytes: 73914809
    num_examples: 161
  - name: humaneval.cpp.bigcode_15b_400m.0.2.reworded
    num_bytes: 68514672
    num_examples: 161
  - name: humaneval.cpp.bigcode_15b_600m.0.2.reworded
    num_bytes: 70059227
    num_examples: 161
  - name: humaneval.cpp.bigcode_15b_800m.0.2.reworded
    num_bytes: 69289473
    num_examples: 161
  - name: humaneval.cpp.codegeex.0.2.reworded
    num_bytes: 70250543
    num_examples: 161
  - name: humaneval.cpp.codegen.0.2.reworded
    num_bytes: 65355449
    num_examples: 161
  - name: humaneval.cpp.cushman001.0.2.reworded
    num_bytes: 6878097
    num_examples: 161
  - name: humaneval.cpp.replit_code.0.2.reworded
    num_bytes: 18647873
    num_examples: 161
  - name: humaneval.cs.bigcode_15b_1000m.0.2.reworded
    num_bytes: 115265463
    num_examples: 158
  - name: humaneval.cs.bigcode_15b_200m.0.2.reworded
    num_bytes: 128116325
    num_examples: 158
  - name: humaneval.cs.bigcode_15b_400m.0.2.reworded
    num_bytes: 116443233
    num_examples: 158
  - name: humaneval.cs.bigcode_15b_600m.0.2.reworded
    num_bytes: 110736924
    num_examples: 158
  - name: humaneval.cs.bigcode_15b_800m.0.2.reworded
    num_bytes: 116921504
    num_examples: 158
  - name: humaneval.cs.codegeex.0.2.reworded
    num_bytes: 108831398
    num_examples: 158
  - name: humaneval.cs.codegen.0.2.reworded
    num_bytes: 115085420
    num_examples: 158
  - name: humaneval.cs.cushman001.0.2.reworded
    num_bytes: 11455476
    num_examples: 158
  - name: humaneval.cs.replit_code.0.2.reworded
    num_bytes: 29978496
    num_examples: 158
  - name: humaneval.d.bigcode_15b_1000m.0.2.reworded
    num_bytes: 69856838
    num_examples: 156
  - name: humaneval.d.bigcode_15b_200m.0.2.reworded
    num_bytes: 69168908
    num_examples: 156
  - name: humaneval.d.bigcode_15b_400m.0.2.reworded
    num_bytes: 66130665
    num_examples: 156
  - name: humaneval.d.bigcode_15b_600m.0.2.reworded
    num_bytes: 60081870
    num_examples: 156
  - name: humaneval.d.bigcode_15b_800m.0.2.reworded
    num_bytes: 68285500
    num_examples: 156
  - name: humaneval.d.codegeex.0.2.reworded
    num_bytes: 67554723
    num_examples: 156
  - name: humaneval.d.codegen.0.2.reworded
    num_bytes: 69538065
    num_examples: 156
  - name: humaneval.d.cushman001.0.2.reworded
    num_bytes: 6543145
    num_examples: 156
  - name: humaneval.d.replit_code.0.2.reworded
    num_bytes: 19332975
    num_examples: 156
  - name: humaneval.go.bigcode_15b_1000m.0.2.reworded
    num_bytes: 75966586
    num_examples: 154
  - name: humaneval.go.bigcode_15b_200m.0.2.reworded
    num_bytes: 90496893
    num_examples: 154
  - name: humaneval.go.bigcode_15b_400m.0.2.reworded
    num_bytes: 80263304
    num_examples: 154
  - name: humaneval.go.bigcode_15b_600m.0.2.reworded
    num_bytes: 80653936
    num_examples: 154
  - name: humaneval.go.bigcode_15b_800m.0.2.reworded
    num_bytes: 79636433
    num_examples: 154
  - name: humaneval.go.codegeex.0.2.reworded
    num_bytes: 74466402
    num_examples: 154
  - name: humaneval.go.codegen.0.2.reworded
    num_bytes: 82565036
    num_examples: 154
  - name: humaneval.go.cushman001.0.2.reworded
    num_bytes: 7919252
    num_examples: 154
  - name: humaneval.go.replit_code.0.2.reworded
    num_bytes: 21740421
    num_examples: 154
  - name: humaneval.jl.bigcode_15b_1000m.0.2.reworded
    num_bytes: 64541752
    num_examples: 159
  - name: humaneval.jl.bigcode_15b_200m.0.2.reworded
    num_bytes: 64272523
    num_examples: 159
  - name: humaneval.jl.bigcode_15b_400m.0.2.reworded
    num_bytes: 84674386
    num_examples: 159
  - name: humaneval.jl.bigcode_15b_600m.0.2.reworded
    num_bytes: 83951098
    num_examples: 159
  - name: humaneval.jl.bigcode_15b_800m.0.2.reworded
    num_bytes: 71891875
    num_examples: 159
  - name: humaneval.jl.codegeex.0.2.reworded
    num_bytes: 49376484
    num_examples: 159
  - name: humaneval.jl.codegen.0.2.reworded
    num_bytes: 49686685
    num_examples: 159
  - name: humaneval.jl.cushman001.0.2.reworded
    num_bytes: 5594623
    num_examples: 159
  - name: humaneval.jl.replit_code.0.2.reworded
    num_bytes: 12432167
    num_examples: 159
  - name: humaneval.js.bigcode_15b_1000m.0.2.reworded
    num_bytes: 63930510
    num_examples: 161
  - name: humaneval.js.bigcode_15b_200m.0.2.reworded
    num_bytes: 71006276
    num_examples: 161
  - name: humaneval.js.bigcode_15b_400m.0.2.reworded
    num_bytes: 71141641
    num_examples: 161
  - name: humaneval.js.bigcode_15b_600m.0.2.reworded
    num_bytes: 66406645
    num_examples: 161
  - name: humaneval.js.bigcode_15b_800m.0.2.reworded
    num_bytes: 65906688
    num_examples: 161
  - name: humaneval.js.codegeex.0.2.reworded
    num_bytes: 68965171
    num_examples: 161
  - name: humaneval.js.codegen.0.2.reworded
    num_bytes: 71850674
    num_examples: 161
  - name: humaneval.js.cushman001.0.2.reworded
    num_bytes: 6756809
    num_examples: 161
  - name: humaneval.js.replit_code.0.2.reworded
    num_bytes: 20658701
    num_examples: 161
  - name: humaneval.lua.bigcode_15b_200m.0.2.reworded
    num_bytes: 56733662
    num_examples: 161
  - name: humaneval.lua.bigcode_15b_400m.0.2.reworded
    num_bytes: 57525953
    num_examples: 161
  - name: humaneval.lua.bigcode_15b_600m.0.2.reworded
    num_bytes: 53575875
    num_examples: 161
  - name: humaneval.lua.bigcode_15b_800m.0.2.reworded
    num_bytes: 54309789
    num_examples: 161
  - name: humaneval.lua.codegeex.0.2.reworded
    num_bytes: 53766400
    num_examples: 161
  - name: humaneval.lua.codegen.0.2.reworded
    num_bytes: 63642889
    num_examples: 161
  - name: humaneval.lua.cushman001.0.2.reworded
    num_bytes: 5726991
    num_examples: 161
  - name: humaneval.lua.replit_code.0.2.reworded
    num_bytes: 14458988
    num_examples: 161
  - name: humaneval.php.bigcode_15b_1000m.0.2.reworded
    num_bytes: 62087493
    num_examples: 161
  - name: humaneval.php.bigcode_15b_200m.0.2.reworded
    num_bytes: 67992787
    num_examples: 161
  - name: humaneval.php.bigcode_15b_400m.0.2.reworded
    num_bytes: 65415347
    num_examples: 161
  - name: humaneval.php.bigcode_15b_600m.0.2.reworded
    num_bytes: 64025429
    num_examples: 161
  - name: humaneval.php.bigcode_15b_800m.0.2.reworded
    num_bytes: 67914229
    num_examples: 161
  - name: humaneval.php.codegeex.0.2.reworded
    num_bytes: 63599818
    num_examples: 161
  - name: humaneval.php.codegen.0.2.reworded
    num_bytes: 71759630
    num_examples: 161
  - name: humaneval.php.cushman001.0.2.reworded
    num_bytes: 6680669
    num_examples: 161
  - name: humaneval.php.replit_code.0.2.reworded
    num_bytes: 18347062
    num_examples: 161
  - name: humaneval.pl.bigcode_15b_1000m.0.2.reworded
    num_bytes: 69839042
    num_examples: 161
  - name: humaneval.pl.bigcode_15b_200m.0.2.reworded
    num_bytes: 79671308
    num_examples: 161
  - name: humaneval.pl.bigcode_15b_400m.0.2.reworded
    num_bytes: 78788842
    num_examples: 161
  - name: humaneval.pl.bigcode_15b_600m.0.2.reworded
    num_bytes: 69916889
    num_examples: 161
  - name: humaneval.pl.bigcode_15b_800m.0.2.reworded
    num_bytes: 73552220
    num_examples: 161
  - name: humaneval.pl.codegeex.0.2.reworded
    num_bytes: 72617126
    num_examples: 161
  - name: humaneval.pl.codegen.0.2.reworded
    num_bytes: 74351768
    num_examples: 161
  - name: humaneval.pl.cushman001.0.2.reworded
    num_bytes: 7317844
    num_examples: 161
  - name: humaneval.pl.replit_code.0.2.reworded
    num_bytes: 23014112
    num_examples: 161
  - name: humaneval.py.bigcode_15b_200m.0.8.reworded
    num_bytes: 55679581
    num_examples: 161
  - name: humaneval.py.bigcode_15b_400m.0.8.reworded
    num_bytes: 49813429
    num_examples: 161
  - name: humaneval.py.replit_code.0.2.reworded
    num_bytes: 16222771
    num_examples: 161
  - name: humaneval.py.starcoder.0.8.reworded
    num_bytes: 50428866
    num_examples: 161
  - name: humaneval.py.starcoderprompted.0.1.reworded
    num_bytes: 53971758
    num_examples: 161
  - name: humaneval.rb.bigcode_15b_1000m.0.2.reworded
    num_bytes: 67446763
    num_examples: 161
  - name: humaneval.rb.bigcode_15b_200m.0.2.reworded
    num_bytes: 70571683
    num_examples: 161
  - name: humaneval.rb.bigcode_15b_400m.0.2.reworded
    num_bytes: 67565830
    num_examples: 161
  - name: humaneval.rb.bigcode_15b_600m.0.2.reworded
    num_bytes: 71419194
    num_examples: 161
  - name: humaneval.rb.bigcode_15b_800m.0.2.reworded
    num_bytes: 69995749
    num_examples: 161
  - name: humaneval.rb.codegeex.0.2.reworded
    num_bytes: 63388920
    num_examples: 161
  - name: humaneval.rb.codegen.0.2.reworded
    num_bytes: 68918022
    num_examples: 161
  - name: humaneval.rb.cushman001.0.2.reworded
    num_bytes: 7084615
    num_examples: 161
  - name: humaneval.rb.replit_code.0.2.reworded
    num_bytes: 17797810
    num_examples: 161
  - name: humaneval.r.codegeex.0.2.reworded
    num_bytes: 64172735
    num_examples: 161
  - name: humaneval.r.codegen.0.2.reworded
    num_bytes: 75777642
    num_examples: 161
  - name: humaneval.r.cushman001.0.2.reworded
    num_bytes: 6509329
    num_examples: 161
  - name: humaneval.rkt.bigcode_15b_1000m.0.2.reworded
    num_bytes: 71049799
    num_examples: 161
  - name: humaneval.rkt.bigcode_15b_200m.0.2.reworded
    num_bytes: 72642020
    num_examples: 161
  - name: humaneval.rkt.bigcode_15b_400m.0.2.reworded
    num_bytes: 73564249
    num_examples: 161
  - name: humaneval.rkt.bigcode_15b_600m.0.2.reworded
    num_bytes: 73730273
    num_examples: 161
  - name: humaneval.rkt.codegeex.0.2.reworded
    num_bytes: 70940774
    num_examples: 161
  - name: humaneval.rkt.codegen.0.2.reworded
    num_bytes: 90161741
    num_examples: 161
  - name: humaneval.rkt.cushman001.0.2.reworded
    num_bytes: 6030454
    num_examples: 161
  - name: humaneval.rkt.replit_code.0.2.reworded
    num_bytes: 18423402
    num_examples: 161
  - name: humaneval.r.replit_code.0.2.reworded
    num_bytes: 19677779
    num_examples: 161
  - name: humaneval.rs.bigcode_15b_1000m.0.2.reworded
    num_bytes: 59702550
    num_examples: 156
  - name: humaneval.rs.bigcode_15b_200m.0.2.reworded
    num_bytes: 67759741
    num_examples: 156
  - name: humaneval.rs.bigcode_15b_400m.0.2.reworded
    num_bytes: 68044357
    num_examples: 156
  - name: humaneval.rs.bigcode_15b_600m.0.2.reworded
    num_bytes: 54658037
    num_examples: 156
  - name: humaneval.rs.bigcode_15b_800m.0.2.reworded
    num_bytes: 62854949
    num_examples: 156
  - name: humaneval.rs.codegeex.0.2.reworded
    num_bytes: 67976569
    num_examples: 156
  - name: humaneval.rs.codegen.0.2.reworded
    num_bytes: 82843583
    num_examples: 156
  - name: humaneval.rs.cushman001.0.2.reworded
    num_bytes: 6310774
    num_examples: 156
  - name: humaneval.rs.replit_code.0.2.reworded
    num_bytes: 17624999
    num_examples: 156
  - name: humaneval.scala.bigcode_15b_1000m.0.2.reworded
    num_bytes: 84533532
    num_examples: 160
  - name: humaneval.scala.bigcode_15b_800m.0.2.reworded
    num_bytes: 87023720
    num_examples: 160
  - name: humaneval.scala.codegeex.0.2.reworded
    num_bytes: 79619828
    num_examples: 160
  - name: humaneval.scala.codegen.0.2.reworded
    num_bytes: 128885303
    num_examples: 160
  - name: humaneval.scala.cushman001.0.2.reworded
    num_bytes: 8500865
    num_examples: 160
  - name: humaneval.scala.replit_code.0.2.reworded
    num_bytes: 22458222
    num_examples: 160
  - name: humaneval.sh.bigcode_15b_1000m.0.2.reworded
    num_bytes: 62768941
    num_examples: 158
  - name: humaneval.sh.bigcode_15b_200m.0.2.reworded
    num_bytes: 75630478
    num_examples: 158
  - name: humaneval.sh.bigcode_15b_400m.0.2.reworded
    num_bytes: 77050658
    num_examples: 158
  - name: humaneval.sh.bigcode_15b_600m.0.2.reworded
    num_bytes: 65325746
    num_examples: 158
  - name: humaneval.swift.bigcode_15b_600m.0.2.reworded
    num_bytes: 70424335
    num_examples: 158
  - name: humaneval.scala.bigcode_15b_200m.0.2.reworded
    num_bytes: 89054581
    num_examples: 160
  - name: humaneval.scala.bigcode_15b_400m.0.2.reworded
    num_bytes: 83343360
    num_examples: 160
  - name: humaneval.scala.bigcode_15b_600m.0.2.reworded
    num_bytes: 89752223
    num_examples: 160
  - name: humaneval.sh.bigcode_15b_800m.0.2.reworded
    num_bytes: 66811937
    num_examples: 158
  - name: humaneval.sh.codegeex.0.2.reworded
    num_bytes: 65196768
    num_examples: 158
  - name: humaneval.sh.codegen.0.2.reworded
    num_bytes: 99280481
    num_examples: 158
  - name: humaneval.sh.cushman001.0.2.reworded
    num_bytes: 6237965
    num_examples: 158
  - name: humaneval.sh.replit_code.0.2.reworded
    num_bytes: 18134838
    num_examples: 158
  - name: humaneval.swift.bigcode_15b_1000m.0.2.reworded
    num_bytes: 68129948
    num_examples: 158
  - name: humaneval.swift.bigcode_15b_200m.0.2.reworded
    num_bytes: 76924134
    num_examples: 158
  - name: humaneval.swift.bigcode_15b_400m.0.2.reworded
    num_bytes: 72042977
    num_examples: 158
  - name: humaneval.swift.bigcode_15b_800m.0.2.reworded
    num_bytes: 70027106
    num_examples: 158
  - name: humaneval.swift.codegeex.0.2.reworded
    num_bytes: 73605273
    num_examples: 158
  - name: humaneval.swift.codegen.0.2.reworded
    num_bytes: 76081675
    num_examples: 158
  - name: humaneval.swift.cushman001.0.2.reworded
    num_bytes: 6766506
    num_examples: 158
  - name: humaneval.swift.replit_code.0.2.reworded
    num_bytes: 21605861
    num_examples: 158
  - name: humaneval.ts.bigcode_15b_1000m.0.2.reworded
    num_bytes: 61005831
    num_examples: 159
  - name: humaneval.ts.bigcode_15b_200m.0.2.reworded
    num_bytes: 68875546
    num_examples: 159
  - name: humaneval.ts.bigcode_15b_400m.0.2.reworded
    num_bytes: 62805583
    num_examples: 159
  - name: humaneval.ts.bigcode_15b_600m.0.2.reworded
    num_bytes: 53733690
    num_examples: 159
  - name: humaneval.ts.bigcode_15b_800m.0.2.reworded
    num_bytes: 64371975
    num_examples: 159
  - name: humaneval.ts.codegeex.0.2.reworded
    num_bytes: 58487751
    num_examples: 159
  - name: humaneval.ts.codegen.0.2.reworded
    num_bytes: 69981611
    num_examples: 159
  - name: humaneval.ts.cushman001.0.2.reworded
    num_bytes: 6768589
    num_examples: 159
  - name: humaneval.ts.replit_code.0.2.reworded
    num_bytes: 18208741
    num_examples: 159
  - name: mbpp.py.codegeex.0.2.reworded
    num_bytes: 86906502
    num_examples: 397
  - name: mbpp.py.codegen.0.2.reworded
    num_bytes: 92562493
    num_examples: 397
  - name: mbpp.py.cushman001.0.2.reworded
    num_bytes: 7629346
    num_examples: 397
  - name: mbpp.py.pystarcoder2.0.1.reworded
    num_bytes: 142442817
    num_examples: 397
  - name: mbpp.py.Salesforce_codegen_16B_mono.0.2.reworded
    num_bytes: 86067040
    num_examples: 397
  - name: humaneval.matlab.keep.gpt_35_turbo_0301.0.2.reworded
    num_bytes: 4837906
    num_examples: 161
  download_size: 1286123094
  dataset_size: 14746371358
tags:
- code
pretty_name: MultiPL-E generated programs and execution results
---

# Raw Data from MultiPL-E

**Uploads are a work in progress. If you are interested in a split that is not yet available, please contact a.guha@northeastern.edu.**

This repository contains the raw data -- both completions and executions -- from MultiPL-E that was used to generate several experimental results from the 
MultiPL-E, SantaCoder, and StarCoder papers.

The original MultiPL-E completions and executions are stored in JOSN files. We use [the following script](https://github.com/nuprl/MultiPL-E/blob/main/upload_completions.py-)
to turn each experiment directory into a dataset split and upload to this repository.

Every split is named `base_dataset`.`language`.`model`.`temperature`.`variation`

- `base_dataset` is either `humaneval` or `mbpp`.
- `language` is the file extension of the programming language. E.g., `py` for Python or `sh` for Bash.
- `model` is the name of the model. Some model names used by MultiPL-E:
   - `bigcode_15b_1000m`:  StarCoderBase
   - `bigcode_15b_200m`, `bigcode_15b_400m`, `bigcode_15b_600m`, `bigcode_15b_800m`: the 20%, 40%, 60%, 80% checkpoints for StarCoderBase
  - `pystarcoder2`: StarCoder
   - there are others, just have a look :)
- `temperature`: this is either 0.2 or 0.8
- `variation`: the variation of the MultiPL-E prompt to use, which should be `reworded`. MultiPL-E has several prompt ablations and the `reworded` prompts are the canonical variation.