region:us
This dataset is databricks/databricks-dolly-15k unfiltered and deduped, removing 640 instances of blatant alignment and 14 duplicates.
14357 instructions remain.

clean.py was first ran on https://huggingface.co/datasets/databricks/databricks-dolly-15k/blob/d72c16e4644a463b9c678c71d9440befd4594556/databricks-dolly-15k.jsonl and then dedupe.py was ran on it. renamed to .json not .jsonl

inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to clean.py
