region:us
---
dataset_info:
  features:
  - name: task_id
    dtype: string
  - name: prompt
    dtype: string
  - name: canonical_solution
    dtype: string
  - name: test
    dtype: string
  - name: entry_point
    dtype: string
  - name: signature
    dtype: string
  - name: docstring
    dtype: string
  - name: context
    dtype: string
  - name: instruction
    dtype: string
  splits:
  - name: test
    num_bytes: 335913
    num_examples: 164
  download_size: 161076
  dataset_size: 335913
---
# Instruct HumanEval

## Summary
InstructHumanEval is a modified version of OpenAI HumanEval. For a given prompt, we extracted its signature, its docstring as well as its header to create a flexing
setting which would allow to evaluation instruction-tuned LLM. The delimiters used in the instruction-tuning procedure can be use to build and instruction that would
allow the model to elicit its best capabilities. Here is an example of use

The prompt can be built as follows, depending on the model's instruction tuning delimiters
```python
from datasets import load_dataset
ds = load_dataset("codeparrot/instructhumaneval", split="test", use_auth_token=True)
prompt_0 = "Human\n" + ds[0]["instruction"] + "\nAssistant\n" + ds[0]["context"] 
print(prompt_0)
```
Output
```
Human:
Write a function has_close_elements(numbers: List[float], threshold: float) -> bool to solve the following problem:
Check if in given list of numbers, are any two numbers closer to each other than given threshold.
>>> has_close_elements([1.0, 2.0, 3.0], 0.5)
False
>>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
True 
Assistant:
from typing import List
def has_close_elements(numbers: List[float], threshold: float) -> bool:
```
The model can therefore complete the instruction and yield better results because it fits its training procedure. 

You can also find the code to evaluate models on the dataset in the [BigCode-evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main). The following sections provide more details on the dataset.

## Dataset description
This dataset is a modified version of [OpenAI HumanEval](https://huggingface.co/datasets/openai_humaneval) that is designed to adapt the benchmark
to instruction fine-tuned models. As a matter of fact, HumanEval evaluates the ability to complete a code given its signature, its docstring and 
potentially some auxiliary functions. 

## Dataset construction
In order to build an instruction version of HumanEval we extracted relevant information from the **prompt** column of the original version
- **signature** : this is the signature of the function to complete. It looks like `def function_name(args:type):-> return_type`.
- **docstring** : this is the docstring of the function. It is the text which describes the purpose of the function.
- **context** : this represents every additional information that is provided in order to help the model complete the function. It includes the imports
and the auxiliary functions.
Our idea was to move from the original format of HumanEval
```
<context>
<signature>
<docstring>
```
And build and **instruction** that would be
```
Write a function <signature> to solve the following problem:
<docstring>
```
From this instruction, we can design an evaluation pipeline for instruction fine-tuned languages models.

## Evaluation
Instruction fine-tuned LLM are built by fine-tuning a base LLM on an instruction dataset. This instruction dataset contains several <input, output>
pairs where each represent an instruction submitted by a user together with the right answer to it. These pairs are framed into a multi-turn conversation
with the help of special tokens which design each member of the interaction e.g. Q user_token `Human:`, an assistant_token `Assistant:` and and `end_token` `\n` 
that designates the end of each turn.

### Code completion
In this case, the LLM is provided with the following prompt
```
user_token + <instruction> + <end_token> + <assistant_token> + <context>
```
It is the expected to complete the function to solve the problem formulated by the `instruction`. It is very similar to the original evaluation with the
advantage that it puts the model in the best condition to understand the task that it is asked to solve. The evaluation is done on the part generated after
`<assistant_token>`.


### Docstring to code
This setting is more complicated as it requires to model to account for the information contained in the instruction such as the function signature. The
LLM is provided with the following prompt
```
user_token + <instruction> + <end_token> + <assistant_token>
```
The model has to generate a function with the correct signature that solve adequately the problem. The evaluation is done by identifying the content of the
function in the generation (by search for the right `entry_point`/`function_name`) and concatenating it with the `<context>` provided.

## How to use the dataset
```python
from datasets import load_dataset

ds = load_dataset("codeparrot/instructhumaneval")
```

```
ds
DatasetDict({
    test: Dataset({
        features: ['task_id', 'prompt', 'canonical_solution', 'test', 'entry_point', 'signature', 'docstring', 'context', 'instruction'],
        num_rows: 164
    })
})
```