region:us
---
dataset_info:
  dataset_size: 54170374.219814844
  download_size: 15314618
  features:
  - name: input_ids
    sequence: int32
  - name: attention_mask
    sequence: int8
  - name: labels
    sequence: int64
  - dtype: string
    name: preview
  - dtype: int64
    name: length
  splits:
  - name: train
    num_bytes: 53254784.429814845
    num_examples: 6156
  - name: test
    num_bytes: 915589.79
    num_examples: 97

---
# zh-tw-pythia-ta8000-v1-e1-tr_sg-301-c1024-sbldt1

This dataset is a part of the `zh-tw-llm` project.

* Tokenizer: `zh-tw-pythia-tokenizer-a8000-v1`
* Built with: `sharegpt`
* Rows: `train` `6156`, `test` `97`
* Max length: `1024`
* Full config:
  ```json
  {"build_with": ["sharegpt"], "preview_length": 128, "sort_by": "length-desc", "translations_settings": {"source_dataset": "zetavg/coct-en-zh-tw-translations-twp-300k", "lang_1_key": "en", "lang_2_key": "ch", "templates": ["English: {lang_1}\nChinese: {lang_2}", "Chinese: {lang_2}\nEnglish: {lang_1}"], "use_template": "random", "rows_limit": 300000, "test_size": 100, "test_split_seed": 42}, "sharegpt_settings": {"source_dataset": "zetavg/ShareGPT-Processed", "train_on_inputs": false, "languages": [{"en": 0.4}, "zh_Hant"], "rows_limit": 8000, "test_size": 0.02, "test_split_seed": 42, "test_rows_limit": 100}}
  ```