region:us
This dataset is https://github.com/teknium1/GPTeacher unfiltered, removing 1489 instances of blatant alignment.
23073 instructions remain.

https://github.com/teknium1/GPTeacher/blob/8afcaaa7a11dd980162d861bd6be970f95eb7174/Codegen/codegen-instruct.json

https://github.com/teknium1/GPTeacher/blob/e3b7aba886c6c0c8ad30a650edfa7a3093fbf57c/Instruct/gpt4-instruct-dedupe-only-dataset.json

https://github.com/teknium1/GPTeacher/blob/5b040645528a38bfa81a258e7646f8c92ad7d0dd/Roleplay/roleplay-simple-deduped-roleplay-instruct.json

i combined all of these files above into gpteacher.json and ran clean.py

normal dedupe.py script didn't find any dupes here.

inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to clean.py