license:openrail region:us
---
license: openrail
---

![Change can be sunshine if you let it in. (1).png](https://cdn-uploads.huggingface.co/production/uploads/64c7bfe8ac1016256b69ea02/z130wkpjplXQRC9PIJOZs.png)
#  📔 **DATASET** 
| **Dataset**                |       Class                                                           |        Number of Questions         |
|    -------                 |  -----------------------------------------------------------------    |     ------------------------             |
| **FLAN_CoT(zs)**           | Reasoning 、 MATH 、 ScienceQA 、 Commonsense                          |        8000         |
| **Prm800k**                | Reasoning 、 MATH                                                      |        6713         |
| **ScienceQA**              | ScienceQA                                                              |        5177         |
| **SciBench**               | ScienceQA                                                              |         695        | 
| **ReClor**                 | Reasoning                                                              |      1624           | 
| **TheoremQA**              | Commonsense 、 MATH 、 ScienceQA                                        |         800        | 
| **OpenBookQA**             | Text_Understanding 、 Reasoning 、 Commonsense 、 ScienceQA              |        5957         | 
| **ARB**                    | Reasoning 、 MATH 、 ScienceQA 、 Commonsense 、 Text_Understanding       |       605          | 
| **Openassistant-guanaco**  | Commonsense 、 Text_Understanding 、 Reasoning                           |        802         | 

# 📌 **Methon**
## *Improving the dataset*
In addition to evaluating based on the finetune1 score, we found that previously fine-tuning on the 200,000, 50,000, and 5,000 datasets for 'dolphin' and 'openorca' resulted in scores better than the 200,000 dataset. Therefore, we plan to prioritize testing with a smaller yet high-quality dataset. We will use the 'platypus' dataset and combine it with 'cot' for stratified sampling and optimizing the dataset based on output length.
## *Dataset Format Definition*
Use "instruction、input、output" tend to lean towards guided datasets. In this format, each sample includes an instruction, an input, and an expected output. The instruction provides guidance on how to process the input to generate the output. This format of dataset is often used to train models to perform specific tasks, as they explicitly indicate the operations the model should perform.
  ```
    {
        "input": "",
        "output": "",
        "instruction": ""
    }
   ```

- ### [FLAN_V2 COT(ZS)](https://huggingface.co/datasets/conceptofmind/cot_submix_original/tree/main)
  We only extract the 'zs_opt' from COT and categorize each task.
- ### [OTHER](https://github.com/arielnlee/Platypus/tree/main/data_pipeline)
  Prm800k, ScienceQA, SciBench, ReClor, TheoremQA, OpenBookQA, ARB, and OpenAssistant-Guanaco datasets adopt the same format as Platypus.
  
## *Sampling Algorithms*
Since the flan_v2 cot dataset includes tasks like:
  - cot_esnli
  - cot_strategyqa
  - cot_qasc
  - stream_qed
  - cot_gsm8k
  - cot_ecqa
  - cot_creak
  - stream_aqua

To ensure this dataset contains diverse high-quality data, we first select zs_opt questions. Then, we filter out questions with output lengths exceeding the average length. This step aims to help the model learn richer reasoning steps. After that, we perform stratified sampling. Initially, we attempted stratified sampling before the length-based filtering, but we found that this approach resulted in varying sample sizes, making it challenging to reproduce. Thus, we decided to first filter by length and then perform stratified sampling.

```py
    import json
    import random
    
    
    with open("cot_ORIGINAL.json", "r") as f: 
        abc = json.load(f)
    
    # --- part1 ---
    zsopt_data = [] # "zs_opt"
    for i in abc :
        if i["template_type"] == "zs_opt":
            zsopt_data.append(i)
    
    # --- part2 ---
    output_lengths = [len(i["targets"]) for i in zsopt_data] 
    average_length = sum(output_lengths) / len(output_lengths) # average length
    
    filtered_data = []
    for a in zsopt_data:
        if len(a["targets"]) >= average_length:
            filtered_data.append(a)  # output length need to >= average_length
    
    
    class_counts = {} # Count the number of samples for each class
    for a in filtered_data:
        task_name = a["task_name"]
        if task_name in class_counts:
            class_counts[task_name] += 1
        else:
            class_counts[task_name] = 1
    
    # --- part3 ---
    total_samples = 8000 # we plan to select a total of 8000 samples
    
    sample_ratios = {}
    for task_name, count in class_counts.items():
        sample_ratios[task_name] = count / len(filtered_data)
    
    sample_sizes = {}
    for task_name, sample_ratio in sample_ratios.items():
        sample_sizes[task_name] = round(sample_ratio * total_samples)
    
    
    stratified_samples = {} # Perform stratified sampling for each class
    for task_name, sample_size in sample_sizes.items():
        class_samples = []
        for data in filtered_data:
            if data["task_name"] == task_name:
                class_samples.append(data)
    
        selected_samples = random.sample(class_samples, sample_size)
        stratified_samples[task_name] = selected_samples
    
    
    final_samples = [] # Convert to the specified format
    for task_name, samples in stratified_samples.items():
        for sample in samples:
            final_samples.append(
                {
                    "input": "",  # use ""
                    "output": sample["targets"], # output
                    "instruction": sample["inputs"], # question
                } 
            )
    
    with open("cot_change.json", "w") as f:
        json.dump(final_samples, f, indent=2)

```
# 🏁 **Feature Work** 
Under discussion...