region:us
This dataset is https://github.com/tatsu-lab/stanford_alpaca unfiltered, removing 2095 instances of blatant alignment.
49907 instructions remain.

clean.py was first ran on https://github.com/tatsu-lab/stanford_alpaca/blob/65512697dc67779a6e53c267488aba0ec4d7c02a/alpaca_data.json

normal dedupe.py script didn't find any dupes here.

inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to clean.py