license:cc-by-nc-sa-4.0 region:us
---
license: cc-by-nc-sa-4.0
viewer: false
---
# Dataset Card for CodeFuse-Evol-instruction-66k

<div align='center'>

  ![logo](LOGO.png)


[[中文]](#chinese)    [[English]](#english)

</div>

<a id="english"></a>

## Dataset Description
Evol-instruction-66k data is based on the method mentioned in the paper "WizardCoder: Empowering Code Large Language Models with Evol-Instruct". It enhances the fine-tuning effect of pre-trained code large models by adding complex code instructions.
This data is processed based on an open-source dataset, which can be found at [Evol-Instruct-Code-80k-v1](https://huggingface.co/datasets/nickrosh/Evol-Instruct-Code-80k-v1). The processing includes low-quality filtering, filtering similar data through HumanEval evaluation, etc. After filtering from the original 80k data, we obtain 66k high-quality training and fine-tuning data.


## Creation
<strong>Filtering low-quality data</strong><br>
  (1) Filter data with instruction length less than 10 words or greater than 1000 words；<br>
  (2) Filter data with output length less than 50 words or greater than 2000 words；<br>
  (3) Filter data with no markdown format or multiple markdown formats in the output；<br>
  (4) Filter data with less than 3 lines or more than 100 lines of code in the markdown format；<br>
  (5) Filter data with a description preceding the markdown format containing more than 200 words.<br>
<strong>Filtering data similar to humaneval</strong><br>
  (1) Filter data containing any function name from humaneval；<br>
  (2) Using NLTK to remove stop words and punctuation from the docstring of humaneval, obtain the core words such as "sort array prime", etc. Filter data containing more than 40% of the core words from humaneval.<br>


## Structure
````
Instruction:
Execute a multistage computational task consisting of organizing the provided list of numerals in an ascending sequence and then determining the middle, or median, figure from the resultant ordered sequence.
        data = [12, 30, 4, 18, 24]
Output:
Here's a Python solution:
        ```python
        data = [12, 30, 4, 18, 24]
        # organizing the list in ascending sequence
        data.sort()
        # Finding the middle figure from the sorted sequence
        n = len(data)
        if n % 2 == 0:
            median = (data[n//2 - 1] + data[n//2]) / 2
        else:
            median = data[n//2]
            
        print(f\"Sorted Data: {data}\")
        print(f\"Median: {median}\")
        ```
First, we sort the list in ascending order using the `sort()` function. Then, we find the median. If the length of the list is even, the median is the average of the two middle numbers. If the length is odd, the median is the middle number. In this specific case, because the list length is odd (5), the middle value (i.e., the median) is the 3rd value in the sorted list.
````
<a id="chinese"></a>
## 简介
Evol-instruction-66k数据是根据论文《WizardCoder: Empowering Code Large Language
Models with Evol-Instruct》中提到的方法，通过添加复杂的代码指令来增强预训练代码大模型的微调效果。
该数据是在开源数据集[Evol-Instruct-Code-80k-v1](https://huggingface.co/datasets/nickrosh/Evol-Instruct-Code-80k-v1)基础上对数据进行了一系列处理，包括低质量过滤、HumanEval评测相似数据过滤等，从原始80k数据筛选后得到66k高质量训练微调数据。


## 数据生产过程
<strong>过滤低质量数据</strong><br>
  (1) 过滤instruction长度小于10个单词或者大于1000个单词的数据；<br>
  (2) 过滤output长度小于50个单词或者大于2000个单词的数据；<br>
  (3) 过滤output中无markdown格式或者有多个markdown的数据；<br>
  (4) 过滤markdown格式中代码少于3行或者大于100行的数据；<br>
  (5) 过滤markdown格式前面描述单词大于200个单词的数据。<br>
<strong>过滤与humaneval相似的数据</strong><br>
  (1) 过滤包含humaneval中任意函数名的数据；<br>
  (2) 采用NLTK去除humaneval的docstring中停用词、标点符号后，得到核心词，比如“sort array prime”等，<br>
     过滤包含了humaneval超过40%核心词的数据。<br>

## 数据结构
````
Instruction:
Execute a multistage computational task consisting of organizing the provided list of numerals in an ascending sequence and then determining the middle, or median, figure from the resultant ordered sequence.
        data = [12, 30, 4, 18, 24]
Output:
Here's a Python solution:
        ```python
        data = [12, 30, 4, 18, 24]
        # organizing the list in ascending sequence
        data.sort()
        # Finding the middle figure from the sorted sequence
        n = len(data)
        if n % 2 == 0:
            median = (data[n//2 - 1] + data[n//2]) / 2
        else:
            median = data[n//2]
            
        print(f\"Sorted Data: {data}\")
        print(f\"Median: {median}\")
        ```
First, we sort the list in ascending order using the `sort()` function. Then, we find the median. If the length of the list is even, the median is the average of the two middle numbers. If the length is odd, the median is the middle number. In this specific case, because the list length is odd (5), the middle value (i.e., the median) is the 3rd value in the sorted list.
````