region:us
This dataset is HuggingFaceH4/instruction-dataset unfiltered, removing 10 instances of blatant alignment.
317 instructions remain.

clean.py was first ran on https://huggingface.co/datasets/HuggingFaceH4/instruction-dataset/blob/fc50699d275ed86a072f1166f7390473176719fd/step3-eval.jsonl then renamed to .json not .jsonl

normal dedupe.py script didn't find any dupes here.

inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to clean.py
