task_categories:conversational size_categories:10K<n<100K language:en license:apache-2.0 not-for-all-audiences conversational roleplay custom-format a. arxiv:2308.05884 region:us
Personal Interaction Pairs between People and AI (PIPPA) is a partially synthetic, community contributed and open-source conversational and roleplaying dataset generated from a subset of submitted logs to the Pygmalion project.
@misc{gosling2023pippa,
      title={PIPPA: A Partially Synthetic Conversational Dataset}, 
      author={Tear Gosling and Alpin Dale and Yinhe Zheng},
      year={2023},
      eprint={2308.05884},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
---
license: apache-2.0
task_categories:
- conversational
language:
- en
tags:
- not-for-all-audiences
- conversational
- roleplay
- custom-format
- a.
pretty_name: PIPPA - Personal Interaction Pairs Between People and AI
size_categories:
- 10K<n<100K
viewer: false
---

# PIPPA - Personal Interaction Pairs between People and AI

It's been a long time coming, but we're proud to finally release the public portion of our conversational dataset to the public. **Personal Interaction Pairs between People and AI** (**PIPPA**) is a partially synthetic, community contributed and open-source conversational and roleplaying dataset generated from a subset of submitted logs to the Pygmalion project.

This dataset is a subset of what we have received - it consists only of the valid conversational logs in which the submitter gave consent to redistribute to the public. Furthermore, we have done our best to redact or modify any personal information that could potentially be found within PIPPA. If you have found something within PIPPA which has not been redacted properly, please contact us via. email at `teargosling@pygmalion.chat` or `alpindale@pygmalion.chat` and we'll take care of it for you. You may contact us for any other purpose as well, including yelling at us for when the next model will be released.

**⚠️ CAUTION: PIPPA contains conversations, themes and scenarios which can be considered "not safe for work" (NSFW) and/or heavily disturbing in nature. Models trained purely with PIPPA may have the tendency to generate X-rated output. You have been warned.**
## Dataset Summary

PIPPA consists of just a little more than 1 million lines of dialogue spread out over 26,000 conversations between users of the popular chatbot website "Character.AI" and its large language model, obtained through a large community effort taking place over the course of several months. Tallying shows that over 1,000 unique personas simulating both real and fictional characters are represented within the dataset, allowing PIPPA and LLMs fine-tuned on it to adapt to many different roleplay domains.

The dataset is represented with a JSONL file, with a singular JSON snippet representing one entire conversation. Every snippet contains the following pieces of data: 

- `submission_timestamp`: The Unix timestamp of when this particular conversation was submitted to the project, in milliseconds.
- `categories`: The categories assigned to the character on the Character.AI website, if any were assigned. If no categories were assigned, it will be `null`
- `bot_id`: The unique ID assigned to the specific character which the user was conversing with on the website.
- `bot_name`: The name of the character.
- `bot_greeting`: The introductory line of the character to the user. This is always the first utterance of dialogue in a conversation.
- `bot_definitions`: Contains whatever was typed in the **Definitions** field in the character creator on the website. This usually consists of one or more example conversations between the user and the character designed to steer the model towards emulating the persona correctly. Bot definitions required a separate effort to gather, and thus may not be present for a specific persona - if this is the case, an empty string is provided. Because the defintions were written on Character.AI, this field usually follows Character.AI's unique formatting and should be preprocessed before feeding into any model - please see **Appendix A** of the paper for further details.
- `bot_description`: Contains whatever was typed in the **Description** field in the character creator on the website. It usually consists of a few sentences which gives a brief overview of the character and any important details about them.
- `conversation`: The conversation between the user and the model. This is represented as a list of dictionaries, each dictionary representing a single utterance and containing two key-value pairs: `message`, referring to the utterance itself and `is_human`, which designates whether the dialogue was generated by the user or the LLM.

For further information about PIPPA, please refer to our [published paper](https://arxiv.org/abs/2308.05884) or contact us at the emails listed above. 

## Files

We publish PIPPA in multiple variants, each a singular JSONL file:

- **pippa.jsonl**: The original dataset, almost exactly as submitted to us (barring any modifications resulting from the redaction of personally identifiable information).

- **pippa_deduped.jsonl**: The 'cleaned' version of PIPPA, with duplicate conversations as well as any conversation with less than three turns removed from the dataset. **We recommend using this file.**

- **pippa_metharme.jsonl**: A version of deduped PIPPA which is formatted in a similar way to our [Metharme instructional models](https://huggingface.co/PygmalionAI/metharme-13b), useful as an example to demonstrate how to properly format the PIPPA dataset.

If you are using HuggingFace's `datasets` library, you can choose the file you wish to use by specifying the name of it (without extension) as an argument, like so: `dataset = load_dataset("PygmalionAI/PIPPA", 'pippa_deduped')`. The default value is `pippa_deduped`.

Thank you for your patience, everyone!

## Citation
If you're using our dataset, please consider citing our work:

```bibtex
@misc{gosling2023pippa,
      title={PIPPA: A Partially Synthetic Conversational Dataset}, 
      author={Tear Gosling and Alpin Dale and Yinhe Zheng},
      year={2023},
      eprint={2308.05884},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```
___
Any relationship between the name of this dataset and any public personas is entirely and totally coincidential.