license:openrail region:us
---
license: openrail
---

![Change can be sunshine if you let it in..png](https://cdn-uploads.huggingface.co/production/uploads/64c7bfe8ac1016256b69ea02/T95Zr1eIZsmkZTRQ3kbUj.png)

#  📔 **DATASET** 
| **Dataset**                |       Class                                                           |        Number of Questions         |
|    -------                 |  -----------------------------------------------------------------    |     ------------------------             |
| **FLAN_CoT(zs)**           | Reasoning 、 MATH 、 ScienceQA 、 Commonsense                          |        8000         |
| **Prm800k**                | Reasoning 、 MATH                                                      |        6713         |
| **ScienceQA**              | ScienceQA                                                              |        5177         |
| **SciBench**               | ScienceQA                                                              |         695        | 
| **ReClor**                 | Reasoning                                                              |      1624           | 
| **TheoremQA**              | Commonsense 、 MATH 、 ScienceQA                                        |         800        | 
| **OpenBookQA**             | Text_Understanding 、 Reasoning 、 Commonsense 、 ScienceQA              |        5957         | 
| **ARB**                    | Reasoning 、 MATH 、 ScienceQA 、 Commonsense 、 Text_Understanding       |       605          | 
| **Openassistant-guanaco**  | Commonsense 、 Text_Understanding 、 Reasoning                           |        802         | 
| **SAT**                    | Text_Understanding 、 Reasoning 、 MATH                                  |        426         | 
| **GRE、GMAT**              | Reasoning 、 MATH                                                        |       254          | 
| **AMC、AIME**              | Reasoning 、 MATH                                                        |        1000         | 
| **LSAT**                   | Reasoning 、 LAW                                                       |        1009       | 

# 📌 **Methon**
## *Improving the dataset*
Based on the content of the "Textbooks are all you need" paper, We want to try fine-tuning using advanced questions.
## *Dataset Format Definition*
Use "instruction、input、output" tend to lean towards guided datasets. In this format, each sample includes an instruction, an input, and an expected output. The instruction provides guidance on how to process the input to generate the output. This format of dataset is often used to train models to perform specific tasks, as they explicitly indicate the operations the model should perform.
  ```
    {
        "input": "",
        "output": "",
        "instruction": ""
    }
   ```

- ### [FLAN_V2 COT(ZS)](https://huggingface.co/datasets/conceptofmind/cot_submix_original/tree/main)
  We only extract the 'zs_opt' from COT and categorize each task.
- ### SAT、GRE、GMAT、AMC、AIME、LSAT
  We will configure the input for datasets such as GRE, GMAT, SAT etc. as "Please read the question and options carefully, then select the most appropriate answer and provide the corresponding explanation." Meanwhile, for the math input, it will be set as "Please provide the answer along with a corresponding explanation based on the given question." Moreover, the questions will be arranged in ascending order of difficulty levels. This is done because, according to the ORCA paper, they started training the model using GPT-3.5 and later transitioned to GPT-4. To avoid the student model from acquiring knowledge beyond its scope and thereby delivering suboptimal results, a progressive learning strategy was utilized. This approach was found to be effective, therefore, in datasets like AMC, AIME which have various levels of difficulty, I have arranged them in a way that embodies this gradual, progressive learning technique.

  Furthermore, their question and options are combined to form the instruction, and the label and solution are merged to become the output.

  Lastly, for the LSAT dataset, since it doesn't involve step-by-step processes, the passage is transformed into instruction, while the combination of the question and options serves as the input, and the label represents the output.

- ### [OTHER](https://github.com/arielnlee/Platypus/tree/main/data_pipeline)
  Prm800k, ScienceQA, SciBench, ReClor, TheoremQA, OpenBookQA, ARB, and OpenAssistant-Guanaco datasets adopt the same format as Platypus.
  
## *Sampling Algorithms*
Since the flan_v2 cot dataset includes tasks like:
  - cot_esnli
  - cot_strategyqa
  - cot_qasc
  - stream_qed
  - cot_gsm8k
  - cot_ecqa
  - cot_creak
  - stream_aqua

To ensure this dataset contains diverse high-quality data, we first select zs_opt questions. Then, we filter out questions with output lengths exceeding the average length. This step aims to help the model learn richer reasoning steps. After that, we perform stratified sampling. Initially, we attempted stratified sampling before the length-based filtering, but we found that this approach resulted in varying sample sizes, making it challenging to reproduce. Thus, we decided to first filter by length and then perform stratified sampling.

```py
    import json
    import random
    
    
    with open("cot_ORIGINAL.json", "r") as f: 
        abc = json.load(f)
    
    # --- part1 ---
    zsopt_data = [] # "zs_opt"
    for i in abc :
        if i["template_type"] == "zs_opt":
            zsopt_data.append(i)
    
    # --- part2 ---
    output_lengths = [len(i["targets"]) for i in zsopt_data] 
    average_length = sum(output_lengths) / len(output_lengths) # average length
    
    filtered_data = []
    for a in zsopt_data:
        if len(a["targets"]) >= average_length:
            filtered_data.append(a)  # output length need to >= average_length
    
    
    class_counts = {} # Count the number of samples for each class
    for a in filtered_data:
        task_name = a["task_name"]
        if task_name in class_counts:
            class_counts[task_name] += 1
        else:
            class_counts[task_name] = 1
    
    # --- part3 ---
    total_samples = 8000 # we plan to select a total of 8000 samples
    
    sample_ratios = {}
    for task_name, count in class_counts.items():
        sample_ratios[task_name] = count / len(filtered_data)
    
    sample_sizes = {}
    for task_name, sample_ratio in sample_ratios.items():
        sample_sizes[task_name] = round(sample_ratio * total_samples)
    
    
    stratified_samples = {} # Perform stratified sampling for each class
    for task_name, sample_size in sample_sizes.items():
        class_samples = []
        for data in filtered_data:
            if data["task_name"] == task_name:
                class_samples.append(data)
    
        selected_samples = random.sample(class_samples, sample_size)
        stratified_samples[task_name] = selected_samples
    
    
    final_samples = [] # Convert to the specified format
    for task_name, samples in stratified_samples.items():
        for sample in samples:
            final_samples.append(
                {
                    "input": "",  # use ""
                    "output": sample["targets"], # output
                    "instruction": sample["inputs"], # question
                } 
            )
    
    with open("cot_change.json", "w") as f:
        json.dump(final_samples, f, indent=2)

```
LSAT arranged according to LEVEL

```py
import json

with open("math-json.json", "r", encoding="utf-8") as f:
    data_list = json.load(f)


sorted_data = sorted(data_list, key=lambda x: x["other"]["level"])


output_data = [
    {
        "input": "Please provide the answer along with a corresponding explanation based on the given question.",
        "output": f"{item['answer']},solution:{item['other']['solution']}",
        "instruction": item["question"],
    }
    for item in sorted_data
]

with open("math_convert.json", "w", encoding="utf-8") as output_file:
    json.dump(output_data, output_file, ensure_ascii=False, indent=4)
```