Update README.md

README.md

@@ -11,10 +11,8 @@ tags:
 - evaluate
 - metric
 description: >-
-  This metric implements the evaluation harness for the HumanEval problem
-  solving dataset described in the paper "Evaluating Large Language Models
-  Trained on Code" (https://arxiv.org/abs/2107.03374).
-duplicated_from: evaluate-metric/code_eval
+  This metric implements code evaluation with execution across multiple languages as used in the paper "OctoPack: Instruction Tuning
+  Code Large Language Models" (https://arxiv.org/abs/2308.07124).
 ---
 
 # Metric Card for Code Eval
@@ -23,7 +21,7 @@ duplicated_from: evaluate-metric/code_eval
 
 The CodeEval metric estimates the pass@k metric for code synthesis. 
 
-It implements the evaluation harness for the HumanEval problem solving dataset described in the paper ["Evaluating Large Language Models Trained on Code"](https://arxiv.org/abs/2107.03374).
+It implements the code exection for HumanEvalPack as described in the paper ["OctoPack: Instruction Tuning Code Large Language Model"](https://arxiv.org/abs/2308.07124).
 
 
 ## How to use 
@@ -40,12 +38,16 @@ The Code Eval metric calculates how good are predictions given a set of referenc
 
 `timeout`: The maximum time taken to produce a prediction before it is considered a "timeout". The default value is `3.0` (i.e. 3 seconds).
 
+`language`: Which language to execute the code in. The default value is `python` and alternatives are `javascript`, `java`, `go`, `cpp`, `rust`
+
+`cargo_string`: The cargo installations to perform for Rust. Defaults to some basic packages, see `code_eval_octopack.py`.
+
 ```python
 from evaluate import load
-code_eval = load("code_eval")
+code_eval = load("Muennighoff/code_eval_octopack")
 test_cases = ["assert add(2,3)==5"]
 candidates = [["def add(a,b): return a*b", "def add(a, b): return a+b"]]
-pass_at_k, results = code_eval.compute(references=test_cases, predictions=candidates, k=[1, 2])
+pass_at_k, results = code_eval.compute(references=test_cases, predictions=candidates, k=[1, 2], language="python")
 ```
 
 N.B.
@@ -63,21 +65,16 @@ The Code Eval metric outputs two things:
 
 `results`: a dictionary with granular results of each unit test.
 
-### Values from popular papers
-The [original CODEX paper](https://arxiv.org/pdf/2107.03374.pdf) reported that the CODEX-12B model had a pass@k score of 28.8% at `k=1`, 46.8% at `k=10` and 72.3% at `k=100`. However, since the CODEX model is not open source, it is hard to verify these numbers. 
-
-
-
 ## Examples 
 
 Full match at `k=1`:
 
 ```python
 from evaluate import load
-code_eval = load("code_eval")
+code_eval = load("Muennighoff/code_eval_octopack")
 test_cases = ["assert add(2,3)==5"]
 candidates = [["def add(a, b): return a+b"]]
-pass_at_k, results = code_eval.compute(references=test_cases, predictions=candidates, k=[1])
+pass_at_k, results = code_eval.compute(references=test_cases, predictions=candidates, k=[1], language="python")
 print(pass_at_k)
 {'pass@1': 1.0}
 ```
@@ -86,10 +83,10 @@ No match for k = 1:
 
 ```python
 from evaluate import load
-code_eval = load("code_eval")
+code_eval = load("Muennighoff/code_eval_octopack")
 test_cases = ["assert add(2,3)==5"]
 candidates = [["def add(a,b): return a*b"]]
-pass_at_k, results = code_eval.compute(references=test_cases, predictions=candidates, k=[1])
+pass_at_k, results = code_eval.compute(references=test_cases, predictions=candidates, k=[1], language="python")
 print(pass_at_k)
 {'pass@1': 0.0}
 ```
@@ -98,50 +95,21 @@ Partial match at k=1, full match at k=2:
 
 ```python
 from evaluate import load
-code_eval = load("code_eval")
+code_eval = load("Muennighoff/code_eval_octopack")
 test_cases = ["assert add(2,3)==5"]
 candidates = [["def add(a, b): return a+b", "def add(a,b): return a*b"]]
-pass_at_k, results = code_eval.compute(references=test_cases, predictions=candidates, k=[1, 2])
+pass_at_k, results = code_eval.compute(references=test_cases, predictions=candidates, k=[1, 2], language="python")
 print(pass_at_k)
 {'pass@1': 0.5, 'pass@2': 1.0}
 ```
 
-## Limitations and bias
-
-As per the warning included in the metric code itself:
-> This program exists to execute untrusted model-generated code. Although it is highly unlikely that model-generated code will do something overtly malicious in response to this test suite, model-generated code may act destructively due to a lack of model capability or alignment. Users are strongly encouraged to sandbox this evaluation suite so that it does not perform destructive actions on their host or network. For more  information on how OpenAI sandboxes its code, see the accompanying paper. Once you have read this disclaimer and taken appropriate precautions, uncomment the following line and proceed at your own risk:
-
-More information about the limitations of the code can be found on the [Human Eval Github repository](https://github.com/openai/human-eval).
-
 ## Citation
 
 ```bibtex
-@misc{chen2021evaluating,
-      title={Evaluating Large Language Models Trained on Code},
-      author={Mark Chen and Jerry Tworek and Heewoo Jun and Qiming Yuan \
-and Henrique Ponde de Oliveira Pinto and Jared Kaplan and Harri Edwards \
-and Yuri Burda and Nicholas Joseph and Greg Brockman and Alex Ray \
-and Raul Puri and Gretchen Krueger and Michael Petrov and Heidy Khlaaf \
-and Girish Sastry and Pamela Mishkin and Brooke Chan and Scott Gray \
-and Nick Ryder and Mikhail Pavlov and Alethea Power and Lukasz Kaiser \
-and Mohammad Bavarian and Clemens Winter and Philippe Tillet \
-and Felipe Petroski Such and Dave Cummings and Matthias Plappert \
-and Fotios Chantzis and Elizabeth Barnes and Ariel Herbert-Voss \
-and William Hebgen Guss and Alex Nichol and Alex Paino and Nikolas Tezak \
-and Jie Tang and Igor Babuschkin and Suchir Balaji and Shantanu Jain \
-and William Saunders and Christopher Hesse and Andrew N. Carr \
-and Jan Leike and Josh Achiam and Vedant Misra and Evan Morikawa \
-and Alec Radford and Matthew Knight and Miles Brundage and Mira Murati \
-and Katie Mayer and Peter Welinder and Bob McGrew and Dario Amodei \
-and Sam McCandlish and Ilya Sutskever and Wojciech Zaremba},
-      year={2021},
-      eprint={2107.03374},
-      archivePrefix={arXiv},
-      primaryClass={cs.LG}
+@article{muennighoff2023octopack,
+      title={OctoPack: Instruction Tuning Code Large Language Models}, 
+      author={Niklas Muennighoff and Qian Liu and Armel Zebaze and Qinkai Zheng and Binyuan Hui and Terry Yue Zhuo and Swayam Singh and Xiangru Tang and Leandro von Werra and Shayne Longpre},
+      journal={arXiv preprint arXiv:2308.07124},
+      year={2023}
 }
 ```
-    
-## Further References 
-
-- [Human Eval Github repository](https://github.com/openai/human-eval)
-- [OpenAI Codex website](https://openai.com/blog/openai-codex/)