license:apache-2.0 region:us
---
license: apache-2.0
---

# sharegpt-hyperfiltered-3k

90k sharegpt convos brought down to ~3k (3243) via language filtering, keyword detection, deduping, and regex. Following things were done:

- Deduplication on first message from human
- Remove non-English convos
- Remove censorship, refusals, and alignment
- Remove incorrect/low-quality answers
- Remove creative tasks
  - ChatGPT's creative outputs are very censored and robotic; I think the base model can do better.
- Remove URLs
- Remove cutoffs
- Remove math/reasoning questions
  - It sucks without CoT prompting, so this data should be mixed with better reasoning examples like OpenOrca or Dolphin.
