task_categories:conversational size_categories:10K<n<100K license:cc arxiv:2306.05685 region:us
---
dataset_info:
  features:
  - name: question_id
    dtype: string
  - name: model_a
    dtype: string
  - name: model_b
    dtype: string
  - name: winner
    dtype: string
  - name: judge
    dtype: string
  - name: conversation_a
    list:
    - name: content
      dtype: string
    - name: role
      dtype: string
  - name: conversation_b
    list:
    - name: content
      dtype: string
    - name: role
      dtype: string
  - name: turn
    dtype: int64
  - name: anony
    dtype: bool
  - name: language
    dtype: string
  - name: tstamp
    dtype: float64
  - name: openai_moderation
    struct:
    - name: categories
      struct:
      - name: harassment
        dtype: bool
      - name: harassment/threatening
        dtype: bool
      - name: hate
        dtype: bool
      - name: hate/threatening
        dtype: bool
      - name: self-harm
        dtype: bool
      - name: self-harm/instructions
        dtype: bool
      - name: self-harm/intent
        dtype: bool
      - name: sexual
        dtype: bool
      - name: sexual/minors
        dtype: bool
      - name: violence
        dtype: bool
      - name: violence/graphic
        dtype: bool
    - name: category_scores
      struct:
      - name: harassment
        dtype: float64
      - name: harassment/threatening
        dtype: float64
      - name: hate
        dtype: float64
      - name: hate/threatening
        dtype: float64
      - name: self-harm
        dtype: float64
      - name: self-harm/instructions
        dtype: float64
      - name: self-harm/intent
        dtype: float64
      - name: sexual
        dtype: float64
      - name: sexual/minors
        dtype: float64
      - name: violence
        dtype: float64
      - name: violence/graphic
        dtype: float64
    - name: flagged
      dtype: bool
  - name: toxic_chat_tag
    struct:
    - name: roberta-large
      struct:
      - name: flagged
        dtype: bool
      - name: probability
        dtype: float64
    - name: t5-large
      struct:
      - name: flagged
        dtype: bool
      - name: score
        dtype: float64
  splits:
  - name: train
    num_bytes: 81159839
    num_examples: 33000
  download_size: 41572998
  dataset_size: 81159839
license: cc
task_categories:
- conversational
size_categories:
- 10K<n<100K
extra_gated_prompt: "Disclaimers and Terms\n\
- This dataset contains conversations that may be considered unsafe, offensive, or upsetting. It is not intended for training dialogue agents without applying appropriate filtering measures. We are not responsible for any outputs of the models trained on this dataset.\n\
- Statements or opinions made in this dataset do not reflect the views of researchers or institutions involved in the data collection effort.\n\
- Users of this data are responsible for ensuring its appropriate use, which includes abiding by any applicable laws and regulations.\n\
- Users of this data should adhere to the terms of use for a specific model when using its direct outputs.\n\
- Users of this data agree to not attempt to determine the identity of individuals in this dataset."
---

## Chatbot Arena Conversations Dataset
This dataset contains 33K cleaned conversations with pairwise human preferences.
It is collected from 13K unique IP addresses on the [Chatbot Arena](https://lmsys.org/blog/2023-05-03-arena/) from April to June 2023.
Each sample includes a question ID, two model names, their full conversation text in OpenAI API JSON format, the user vote, the anonymized user ID, the detected language tag, the OpenAI moderation API tag, the additional toxic tag, and the timestamp.

To ensure the safe release of data, we have made our best efforts to remove all conversations that contain personally identifiable information (PII).
User consent is obtained through the "Terms of use" section on the data collection website.
In addition, we have included the OpenAI moderation API output to flag inappropriate conversations. 
However, we have chosen to keep unsafe conversations intact so that researchers can study the safety-related questions associated with LLM usage in real-world scenarios as well as the OpenAI moderation process.
As an example, we included additional toxic tags that are generated by our own toxic tagger, which are trained by fine-tuning T5 and RoBERTa on manually labeled data.

**Basic Statistics**
| Key | Value |
| --- | --- |
| # Conversations | 33,000 |
| # Models | 20 |
| # Users | 13,383 |
| # Languages | 96 |
| Avg. # Turns per Sample | 1.2 |
| Avg. # Tokens per Prompt | 52.3 |
| Avg. # Tokens per Response | 189.5 |


## Uniqueness and Potential Usage
Compared to existing human preference datasets like [Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf), and [OpenAssistant/oasst1](https://huggingface.co/datasets/OpenAssistant/oasst1). This dataset
- Contains the outputs of 20 LLMs including stronger LLMs such as GPT-4 and Claude-v1. It also contains many failure cases of these state-of-the-art models.
- Contains unrestricted conversations from over 13K users in the wild.

We believe it will help the AI research community answer important questions around topics like:
- Characteristics and distributions of real-world user prompts
- Training instruction-following models
- Improve and evaluate LLM evaluation methods
- Model selection and request dispatching algorithms
- AI safety and content moderation

## Disclaimers and Terms
- **This dataset contains conversations that may be considered unsafe, offensive, or upsetting.** It is not intended for training dialogue agents without applying appropriate filtering measures. We are not responsible for any outputs of the models trained on this dataset.
- Statements or opinions made in this dataset do not reflect the views of researchers or institutions involved in the data collection effort.
- Users of this data are responsible for ensuring its appropriate use, which includes abiding by any applicable laws and regulations.
- Users of this data should adhere to the terms of use for a specific model when using its direct outputs.
- Users of this data agree to not attempt to determine the identity of individuals in this dataset.

## Visualization and Elo Rating Calculation
This Colab [notebook](https://colab.research.google.com/drive/1J2Wf7sxc9SVmGnSX_lImhT246pxNVZip?usp=sharing) provides some visualizations and shows how to compute Elo ratings with the dataset.

## License
The user prompts are licensed under CC-BY-4.0, while the model outputs are licensed under CC-BY-NC-4.0.

## Citation
```
@misc{zheng2023judging,
      title={Judging LLM-as-a-judge with MT-Bench and Chatbot Arena}, 
      author={Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric. P Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica},
      year={2023},
      eprint={2306.05685},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```