license:other region:us
---
license: other
---

## Overview

This dataset is mostly a continuation of https://hf.co/datasets/jondurbin/airoboros-2.1, with some notable additions and fixes.

- Some of the content is "toxic"/"harmful", and contains profanity and other types of sensitive content.
- None of the content or views contained in text within this dataset necessarily align with my personal beliefs or opinions, they are simply text generated by LLMs and/or scraped from the web.
- Use with caution, particularly in locations with less-than-free speech laws.
- You, and you alone are responsible for having downloaded the dataset and having a copy of the contents therein and I am completely indemnified from any and all liabilities.

### 2.1 Contamination

I accidentally included some of the benchmark data in the first version of the airboros-2.1 model, which is why it had a crazy high truthfulqa score.  Discussions here:
- https://huggingface.co/jondurbin/airoboros-l2-70b-2.1/discussions/3#64f325ce352152814d1f796a
- https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/225#64f0997659da193a12b78c32

I flagged it for removal and recreated the model right away, but the leaderboard cached the old results so it took some time to reflect.

Some of the instructors I use create overlapping data, and it's hard to filter, especially since the instructions aren't typically verbatim with the benchmark questions.

This time around, I used `thenlper/gte-small` to calculate embeddings of the instructions, along with a faiss index, and removed anything from the dataset that had a similarity score < 0.15 (from truthfulqa).  If you have a better way of checking, please let me know!

I haven't done the same for most other benchmarks (yet) because there are hundreds of thousands of instructions and it would be pretty computationally expensive to do.  That said, I only have ~1279 multiple choice questions, all randomly GPT generated, so there's probably little-to-no overlap.

### Awareness

I added a new "awareness" instructor, which aims to add a lot more nuance to responses relating to time, location, senses, etc. based on the system prompt.

For example, if you are using the standard prompt with user/assistant, and ask how long it would take to get to Chicago, the answer will be something about AI not having a physical presence.
If, on the other hand, you are using a system prompt with a human character specified, the model attempts to infer location from "home" and will provide a more nuanced answer as a human would (in theory).

https://github.com/jondurbin/airoboros/commit/e91562c88d7610edb051606622e7c25a99884f7e

### Editor

I created a text edit instructor as well, which uses a reverse prompt mechanism, meaning it takes the existing writing samples that have been generated, rewrites them to have misspellings, poor grammar, etc., then uses a prompt like "Please correct and improve the text." with the original well-written text and target output.

https://github.com/jondurbin/airoboros/commit/e60a68de5f9622320c9cfff3b238bd83cc7e373b

### Writing

I regenerated (almost) all of the training data that included "Once upon a time..." because it's too cliche and boring.

### Multiple choice

I created many more multiple choice questions, many of which have additional text context.

### Roleplay/conversation

I re-created all of the GTKM and RP datasets this time around, removing all of the "USER: " and "ASSISTANT: " prefixes from the instructions/responses, so it's more compatible with existing interfaces.

The GTKM instructor now does the same thing as RP, in that it saves each round of "conversation" as a separate row in the output - previously it only saved the final response, which may not have been sufficient since I don't typically train on inputs.

### UTF-8 to ASCII

I replaced most of the "standard" utf-8 sequences - left double quote, right double quote, left apostraphe, ellipses - with standard ascii characters.  I don't know if this was contributing to part of the issue with eos tokens being produced after apostraphes, but I figured it was worth trying.

### Summarization

I also included 500 examples from:
https://hf.co/datasets/mattpscott/airoboros-summarization

These are existing summarizarions from various public datasets, formatted to airoboros style contextual qa.

Thanks Matt!


### Usage/license info

Much (most) of the data was generated via gpt-4 API calls, which has a restriction in the ToS about "competing" models.  Please seek legal advice if you plan to build or use a model that includes this dataset in a commercial setting.