license:apache-2.0 region:us
---
dataset_info:
  features:
  - name: message_id
    dtype: string
  - name: parent_id
    dtype: string
  - name: user_id
    dtype: string
  - name: created_date
    dtype: string
  - name: text
    dtype: string
  - name: role
    dtype: string
  - name: lang
    dtype: string
  - name: review_count
    dtype: int32
  - name: review_result
    dtype: bool
  - name: deleted
    dtype: bool
  - name: rank
    dtype: float64
  - name: synthetic
    dtype: bool
  - name: model_name
    dtype: 'null'
  - name: detoxify
    struct:
    - name: identity_attack
      dtype: float64
    - name: insult
      dtype: float64
    - name: obscene
      dtype: float64
    - name: severe_toxicity
      dtype: float64
    - name: sexual_explicit
      dtype: float64
    - name: threat
      dtype: float64
    - name: toxicity
      dtype: float64
  - name: message_tree_id
    dtype: string
  - name: tree_state
    dtype: string
  - name: emojis
    struct:
    - name: count
      sequence: int32
    - name: name
      sequence: string
  - name: labels
    struct:
    - name: count
      sequence: int32
    - name: name
      sequence: string
    - name: value
      sequence: float64
  - name: parent_text
    dtype: string
  - name: spam
    dtype: float64
  - name: fails_task
    dtype: float64
  - name: lang_mismatch
    dtype: float64
  - name: pii
    dtype: float64
  - name: not_appropriate
    dtype: float64
  - name: hate_speech
    dtype: float64
  - name: sexual_content
    dtype: float64
  - name: quality
    dtype: float64
  - name: toxicity
    dtype: float64
  - name: humor
    dtype: float64
  - name: helpfulness
    dtype: float64
  - name: creativity
    dtype: float64
  - name: violence
    dtype: float64
  splits:
  - name: train
    num_bytes: 59657796
    num_examples: 34059
  - name: validation
    num_bytes: 3164029
    num_examples: 1816
  download_size: 25173939
  dataset_size: 62821825
license: apache-2.0
---
# Dataset Card for "oasst1_dense_flat"



[OASST1 dataset](https://huggingface.co/datasets/OpenAssistant/oasst1)
But where with retrieved parent_text, and where we only keep messages with dense annotations (all labels have 2 annotators)

```python
from datasets import Dataset, DatasetDict

d={}
for split in ['train','validation']:
    df=load_dataset("OpenAssistant/oasst1")[split].to_pandas()
    m2t=df.set_index("message_id")['text'].to_dict()
    df['parent_text']=df.parent_id.map(lambda x: m2t.get(x,''))

    df=df[df.labels.map(lambda x:x!=None)]
    df=df[df.labels.map(lambda x:x['count'].min()>2)]

    labels=df.labels.map(lambda x:list(x['name'])).value_counts().index[0]
    df=df[df.labels.map(lambda x:x!=None)]
    df=df[df.labels.map(lambda x:list(x['name'])==labels)]
    for label in labels:
        df[label]=df.labels.map(lambda x: x['value'][list(x['name']).index(label)])
    d[split]=Dataset.from_pandas(df,preserve_index=False)
    
DatasetDict(d).push_to_hub('oasst1_dense_flat')
```
https://github.com/LAION-AI/Open-Assistant

```
@article{kopf2023openassistant,
  title={OpenAssistant Conversations--Democratizing Large Language Model Alignment},
  author={K{\"o}pf, Andreas and Kilcher, Yannic and von R{\"u}tte, Dimitri and Anagnostidis, Sotiris and Tam, Zhi-Rui and Stevens, Keith and Barhoum, Abdullah and Duc, Nguyen Minh and Stanley, Oliver and Nagyfi, Rich{\'a}rd and others},
  journal={arXiv preprint arXiv:2304.07327},
  year={2023}
}
```