size_categories:1K<n<10K language:de license:apache-2.0 arxiv:2304.07327 region:us
---
dataset_info:
  features:
  - name: conversation
    list:
    - name: role
      dtype: string
    - name: text
      dtype: string
  splits:
  - name: train
    num_bytes: 8022604.792326268
    num_examples: 3721
  download_size: 4325950
  dataset_size: 8022604.792326268
license: apache-2.0
language:
- de
size_categories:
- 1K<n<10K
---
# German OpenAssistant Conversations Dataset (OASST-DE)
With the goal of advancing open-source, german-language LLM research, we present 
OASST-DE: a high quality subset of a recent (25.08.23) dump from the [OpenAssistant website](https://www.open-assistant.io/)
translated to German using the GPT-3.5 API. More details on how the dataset was filtered and translated under [dataset creation.](#dataset-creation-process)
For more details on the OpenAssistant Project, look at the [first OASST dataset (OASST1)](https://huggingface.co/datasets/OpenAssistant/oasst1), [the Open-Assistant GitHub repo](https://github.com/LAION-AI/Open-Assistant)
or [our paper](https://arxiv.org/abs/2304.07327).

This dataset was created as part of LAION's LeoLM (Linguistically Enhanced Open Language Model) project led by Björn Plüster. 
Check out LeoLM-Chat trained with OASST-DE ([7b](https://huggingface.co/LeoLM/leo-hessianai-7b-chat), [13b](https://huggingface.co/LeoLM/leo-hessianai-13b-chat)) finetuned on OASST-DE and read [their blog post](https://laion.ai/blog/leo-lm/)) for more info on LeoLM.

## Dataset Creation Process
This dataset was created from a recent OASST dump by following these steps:
- Filter for Top1 response trees with assistant response leaves
- Filter first prompt quality >= 0.5
- Filter total conversation length < 1900 tokens to fit in GPT3.5 context length
- Filter for `'lang' == 'de'` -> add to dataset
- Filter for `'lang' == 'en'` (other languages often result in failed translations)
- Translate using GPT-3.5-turbo API (total cost ~15$).

This results in around 3.7k samples of high-quality assistant conversations.

## Dataset Structure
This dataset has only one `'conversation'` field. Each example is a list of an alternating conversation between `'prompter'` and `'assistant'`,
where each entry is a dict with `'text'` and `'role'` fields:
```json
"conversation": [
  {"role": "prompter", "text": "Moin, wie geht's dir?"},
  {"role": "assistant", "text": "Moin Moin! Mir geht es gut, und dir?"},
  ...
]
```

## Usage with 🤗Datasets:
```python
from datasets import load_dataset
ds = load_dataset("OpenAssistant/oasst_de", split="train")
print(ds[0]["conversation"])
```