task_categories:conversational size_categories:10K<n<100K language:vi license:apache-2.0 region:us
---
license: apache-2.0
dataset_info:
  features:
  - name: message_id
    dtype: string
  - name: parent_id
    dtype: string
  - name: user_id
    dtype: string
  - name: created_date
    dtype: string
  - name: text
    dtype: string
  - name: role
    dtype: string
  - name: lang
    dtype: string
  - name: review_count
    dtype: int32
  - name: review_result
    dtype: bool
  - name: deleted
    dtype: bool
  - name: rank
    dtype: int32
  - name: synthetic
    dtype: bool
  - name: model_name
    dtype: string
  - name: detoxify
    struct:
    - name: toxicity
      dtype: float64
    - name: severe_toxicity
      dtype: float64
    - name: obscene
      dtype: float64
    - name: identity_attack
      dtype: float64
    - name: insult
      dtype: float64
    - name: threat
      dtype: float64
    - name: sexual_explicit
      dtype: float64
  - name: message_tree_id
    dtype: string
  - name: tree_state
    dtype: string
  - name: emojis
    sequence:
    - name: name
      dtype: string
    - name: count
      dtype: int32
  - name: labels
    sequence:
    - name: name
      dtype: string
    - name: value
      dtype: float64
    - name: count
      dtype: int32
  - name: text_chunks
    sequence: string
  - name: text_translation
    dtype: string
  splits:
  - name: train
    num_bytes: 59922108.85834358
    num_examples: 38537
  download_size: 39428167
  dataset_size: 59922108.85834358
task_categories:
- conversational
language:
- vi
size_categories:
- 10K<n<100K
---

This dataset contains vi subsets (first 191 examples) and auto-translation from en to vi subsets (the rest, 38346 examples) from [OASST1](https://huggingface.co/datasets/OpenAssistant/oasst1). All auto-translation examples are generated using [VietAI envit5-translation](https://huggingface.co/VietAI/envit5-translation).
The vi subsets have the same features as the original dataset. Meanwhile, the auto-translation subsets introduce two new features:   
- `"text_chunks"` is a list that contains chunked text split from `"text"`, each chunk has no more than 300 tokens. The sent_tokenizer and word_tokenzier used are from spacy en_core_web_sm model.
- `"text_translation"` contains merged of all translated chunks. Due to the auto-translation model, all new-line symbols (`\n`) are removed.   
The translation script can be found at `translate_en_to_vi.py`