region:us
---
dataset_info:
  features:
  - name: conversation
    struct:
    - name: samples
      list:
      - name: ASSISTANT
        dtype: string
      - name: USER
        dtype: string
  - name: source
    dtype: string
  splits:
  - name: train
    num_bytes: 376370658
    num_examples: 66951
  download_size: 88693772
  dataset_size: 376370658
---
## Megacode-best

Megacode-best is a filtered and deduped version of [megacode-2 dataset](https://huggingface.co/datasets/rombodawg/2XUNCENSORED_MegaCodeTraining188k). In my analysis, I found many similar instruction in the original dataset which I wanted to filter out to avoid overfitting and improve generalisation. 
Filtering technique
1. GTE-base embeddings + Cosine similarity deduplication 

GTE-base was chosen over bge-base models because GTE-models are trained on 20M code tokens and showed better results in similarity search. 

The total number of samples was reduced to 66k which is almost 1/3rd of the original dataset size. 

This dataset was used to train the latest [Open-assistant code llama 2](https://huggingface.co/OpenAssistant/codellama-13b-oasst-sft-v10)
