region:us
This dataset is https://github.com/yizhongw/self-instruct unfiltered and deduped, removing 1600 instances of blatant alignment and 26 duplicates.
80813 instructions remain.

clean.py was first ran on https://github.com/yizhongw/self-instruct/blob/0b26ccaa415992100fa32df62d41b994cf928e23/data/gpt3_generations/batch_221203/all_instances_82K.jsonl and then dedupe.py was ran on it, renamed to .json not .jsonl

inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to clean.py