task_categories:text-generation language:zh region:us
---
dataset_info:
  features:
  - name: line
    dtype: string
  splits:
  - name: train
    num_bytes: 18198166302
    num_examples: 85165683
  download_size: 13296002208
  dataset_size: 18198166302
task_categories:
- text-generation
language:
- zh
---
# CC-100 zh-Hant (Traditional Chinese)

From https://data.statmt.org/cc-100/, only zh-Hant - Chinese (Traditional). Broken into lines, with each line as a row.

Estimated to have around 4B tokens when tokenized with the [`bigscience/bloom`](https://huggingface.co/bigscience/bloom) tokenizer.

There's another version that the text is split by paragraphs instead of lines: [`zetavg/CC-100-zh-Hant-merged`](https://huggingface.co/datasets/zetavg/CC-100-zh-Hant-merged).

## References

Please cite the following if you found the resources in the CC-100 corpus useful.

* **Unsupervised Cross-lingual Representation Learning at Scale**, *Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov*, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), p. 8440-8451, July 2020, [pdf](https://www.aclweb.org/anthology/2020.acl-main.747.pdf), [bib](https://www.aclweb.org/anthology/2020.acl-main.747.bib) .
* **CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data**, *Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, Edouard Grave*, Proceedings of the 12th Language Resources and Evaluation Conference (LREC), p. 4003-4012, May 2020,  [pdf](https://www.aclweb.org/anthology/2020.lrec-1.494.pdf), [bib](https://www.aclweb.org/anthology/2020.lrec-1.494.bib).