task_categories:text-generation size_categories:n>1T language:zh region:us
WuDaoCorpora Text is a large pretraining Chinese corpus constructed by Beijing Academy of Artificial Intelligence(BAAI). The total data volume of the dataset has exceeded 5TB, including 200GB open data.


Compared with other pretraining corpora, the WuDaoCorpora Text has the following advantages.

1) In the process of data collection, we classify the quality of web pages according to the proportion of words in web pages and the integrity of DOM trees, and select high-quality web page for data collection to ensure the corpus quality.

2) Through data cooperation with other institutions and web page data crawling, the dataset covers a wide range types of Chinese text, including news, comments, encyclopedias, forums, blogs, academic papers, etc.

3) The dataset uses more than 20 cleaning rules to obtain the final corpus from the 100TB original web page data. In the cleaning process, special attention is paid to the removal of private information to avoid the risk of privacy disclosure.

4) The dataset contains 50+ data tags, such as education and laws, which is convenient for users to extract specific-domain data for model training in that field.


Please obey the following agreement if you use our dataset.

https://data.baai.ac.cn/resources/agreement/BAAIDataAgreement.pdf
@misc{ c6a3fe684227415a9db8e21bac4a15ab,
  author       = {Zhao Xue and Hanyu Zhao and Sha Yuan and Yequan Wang},
  title        = {{WuDaoCorpora Text}},
  year         = 2022,
  month        = dec,
  publisher    = {Science Data Bank},
  version      = {V1},
  doi          = {10.57760/sciencedb.o00126.00004},
  url          = https://doi.org/10.57760/sciencedb.o00126.00004
}
---
language:
- zh
task_categories:
- text-generation
size_categories:
- n>1T
---

# 悟道(WuDao)資料集
非原製作者，僅搬移。

此資料集下載約60GB，解壓縮後約220GB。

### 原始連結
[Science Data Bank](https://www.scidb.cn/en/detail?dataSetId=c6a3fe684227415a9db8e21bac4a15ab)

## 使用
```bash
pip install patool wget opencc
```
```python
from datasets import load_dataset
# 簡中
load_dataset("p208p2002/wudao",streaming=True,split="zhs")
# 繁中 (使用opencc轉換)
load_dataset("p208p2002/wudao",streaming=True,split="zht")
```

## 清除資料
當下載失敗的時候請手動清除資料
```bash
rm -rf ~/.cache/wudao_dataset
```

## 資料類別統計
```json
{
  "_total": 59100001,
  "豆瓣话题": 209027,
  "科技": 1278068,
  "经济": 1096215,
  "汽车": 1368193,
  "娱乐": 1581947,
  "农业": 1129758,
  "军事": 420949,
  "社会": 446228,
  "游戏": 754703,
  "教育": 1133453,
  "体育": 660858,
  "旅行": 821573,
  "国际": 630386,
  "房产": 387786,
  "文化": 710648,
  "法律": 36585,
  "股票": 1205,
  "博客": 15467790,
  "日报": 16971,
  "评论": 13867,
  "孕育常识": 48291,
  "健康": 15291,
  "财经": 54656,
  "医学问答": 314771,
  "资讯": 1066180,
  "科普文章": 60581,
  "百科": 27273280,
  "酒业": 287,
  "经验": 609195,
  "新闻": 846810,
  "小红书攻略": 185379,
  "生活": 23,
  "网页文本": 115830,
  "观点": 1268,
  "海外": 4,
  "户外": 5,
  "美容": 7,
  "理论": 247,
  "天气": 540,
  "文旅": 2999,
  "信托": 62,
  "保险": 70,
  "水利资讯": 17,
  "时尚": 1123,
  "亲子": 39,
  "百家号文章": 335591,
  "黄金": 216,
  "党建": 1,
  "期货": 330,
  "快讯": 41,
  "国内": 15,
  "国学": 614,
  "公益": 15,
  "能源": 7,
  "创新": 6
}
```

## Cite
```
@misc{ c6a3fe684227415a9db8e21bac4a15ab,
  author       = {Zhao Xue and Hanyu Zhao and Sha Yuan and Yequan Wang},
  title        = {{WuDaoCorpora Text}},
  year         = 2022,
  month        = dec,
  publisher    = {Science Data Bank},
  version      = {V1},
  doi          = {10.57760/sciencedb.o00126.00004},
  url          = https://doi.org/10.57760/sciencedb.o00126.00004
}
```