region:us
---
dataset_info:
  features:
  - name: cluster
    struct:
    - name: samples
      list:
      - name: input
        dtype: string
      - name: output
        dtype: string
  - name: source
    dtype: string
  - name: instruction
    dtype: string
  - name: num_samples
    dtype: int64
  splits:
  - name: train
    num_bytes: 900092818
    num_examples: 328906
  download_size: 462629849
  dataset_size: 900092818
---
## Best of Orca

This is a filtered version of Orca GPT4 1M instructions. From repeated experiments and analysis, I came to the conclusion that original dataset
contains a lot of low-quality instructions which contributes to only poor generalization. 
The solution I came up with is to filter the dataset and remove the unwanted samples. I applied two levels of filters
1. Removed instructions with less than 100 tokens in response. 
2. Data deduplication grouped by instruction type using GTE embedding and cosine similarity (threshold>0.95)

After these two steps, the number of samples was reduced to 1/3rd of the original count. 
For selecting a sample from each cluster, I tried different methods including random selection from a cluster.

We used this dataset to train multiple Open-Assistant models to confirm my hypothesis that data quality matter more than quantity. 
This dataset was used in some of our models best models including https://huggingface.co/OpenAssistant/llama2-70b-oasst-sft-v10

⭐️ All models perform much better than models trained on full ORCA samples. 

## Credits
* This wouldn't be possible without the amazing work of Eric in recreating the ORCA dataset. Check it out:
https://huggingface.co/datasets/ehartford/dolphin
* This dataset was created in association with the Open-Assistant team @jordanclive and @andreaskoepf

## Citations
```
@misc{Orca-best,
  title = {Orca-best: A filtered version of orca gpt4 dataset.},
  author = {Shahul Es},
  year = {2023},
  publisher = {HuggingFace},
  journal = {HuggingFace repository},
  howpublished = {\url{https://huggingface.co/datasets/shahules786/orca-best/},
}
```