size_categories:1K<n<10K language:en region:us
---
language:
- en
size_categories:
- 1K<n<10K
---
This dataset is Anthropic/hh-rlhf unfiltered and deduped, removing 18170 instances of blatant alignment and 51954 duplicates.
99228 instructions remain.

i downloaded all the data (except red team) from https://huggingface.co/datasets/Anthropic/hh-rlhf/tree/09be8c5bbc57cb3887f3a9732ad6aa7ec602a1fa unarchived it merged it all into one file then ran clean.py and then ran dedupe.py to get the resulting file

inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to clean.py