region:us
This dataset is HuggingFaceH4/asss unfiltered and deduped, removing 29 instances of blatant alignment and 21 duplicates.
398 instructions remain.

clean.py was first ran on https://huggingface.co/datasets/HuggingFaceH4/asss/blob/122c06526f06a25c54614c1f5c63b4c3f51e96fb/surge-scale-si-aws-dataset.jsonl and then dedupe.py was ran on it. renamed to .json not .jsonl

inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to clean.py