region:us
This dataset is the ShareGPT unfiltered dataset anon8231489123/ShareGPT_Vicuna_unfiltered, removing instances of blatant alignment and removes duplicates.
33714 instructions remain.

clean.py was first ran on hakurei/open-instruct-v1/subsets/sharegpt_data.json and then dedupe.py was ran on it.

inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to sharegpt_clean.py