File size: 7,298 Bytes
3afd260 ae16132 e6b6048 3afd260 e6b6048 ae16132 0d0a80c ae16132 e6b6048 3afd260 e6b6048 3afd260 a7c7981 e6b6048 a7c7981 ae16132 3afd260 e6b6048 3afd260 e6b6048 3afd260 e6b6048 3afd260 e6b6048 6f533f1 3afd260 e6b6048 3afd260 e6b6048 6f533f1 3afd260 e6b6048 5e219a4 e6b6048 3afd260 e6b6048 41c7de1 3afd260 e6b6048 3afd260 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 |
---
language: zh
datasets: CLUECorpusSmall
widget:
- text: "北京是[MASK]国的首都。"
---
# Chinese Xlarge Whole Word Masking RoBERTa Model
## Model description
This is an xlarge Chinese Whole Word Masking RoBERTa model pre-trained by [TencentPretrain](https://github.com/Tencent/TencentPretrain) introduced in [this paper](https://arxiv.org/abs/2212.06385), which inherits [UER-py](https://github.com/dbiir/UER-py/) to support models with parameters above one billion, and extends it to a multimodal pre-training framework.
In order to facilitate users in reproducing the results, we used a publicly available corpus and word segmentation tool, and provided all training details.
You can download the model either from the [UER-py Modelzoo page](https://github.com/dbiir/UER-py/wiki/Modelzoo), or via HuggingFace from the link [roberta-xlarge-wwm-chinese-cluecorpussmall](https://huggingface.co/uer/roberta-xlarge-wwm-chinese-cluecorpussmall):
## How to use
You can use this model directly with a pipeline for masked language modeling:
```python
>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='uer/roberta-xlarge-wwm-chinese-cluecorpussmall')
>>> unmasker("北京是[MASK]国的首都。")
[
{'score': 0.9298505783081055,
'token': 704,
'token_str': '中',
'sequence': '北 京 是 中 国 的 首 都 。'},
{'score': 0.05041525512933731,
'token': 2769,
'token_str': '我',
'sequence': '北 京 是 我 国 的 首 都 。'},
{'score': 0.004921116400510073,
'token': 4862,
'token_str': '祖',
'sequence': '北 京 是 祖 国 的 首 都 。'},
{'score': 0.0020684923510998487,
'token': 3696,
'token_str': '民',
'sequence': '北 京 是 民 国 的 首 都 。'},
{'score': 0.0018144999630749226,
'token': 3926,
'token_str': '清',
'sequence': '北 京 是 清 国 的 首 都 。'}
]
```
Here is how to use this model to get the features of a given text in PyTorch:
```python
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('uer/roberta-xlarge-wwm-chinese-cluecorpussmall')
model = BertModel.from_pretrained("uer/roberta-xlarge-wwm-chinese-cluecorpussmall")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```
and in TensorFlow:
```python
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('uer/roberta-xlarge-wwm-chinese-cluecorpussmall')
model = TFBertModel.from_pretrained("uer/roberta-xlarge-wwm-chinese-cluecorpussmall")
text = "用你喜欢的任何文本替换我。"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```
## Training data
[CLUECorpusSmall](https://github.com/CLUEbenchmark/CLUECorpus2020/) is used as training data.
## Training procedure
Models are pre-trained by [TencentPretrain](https://github.com/Tencent/TencentPretrain) on [Tencent Cloud](https://cloud.tencent.com/). We pre-train 500,000 steps with a sequence length of 128 and then pre-train 250,000 additional steps with a sequence length of 512.
[jieba](https://github.com/fxsjy/jieba) is used as word segmentation tool.
Stage1:
```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
--vocab_path models/google_zh_vocab.txt \
--dataset_path cluecorpussmall_seq128_dataset.pt \
--processes_num 32 --seq_length 128 \
--dynamic_masking --data_processor mlm
```
```
deepspeed pretrain.py --deepspeed --deepspeed_config models/deepspeed_config.json --dataset_path cluecorpussmall_seq128_dataset.pt \
--vocab_path models/google_zh_vocab.txt \
--config_path models/bert/xlarge_config.json \
--output_model_path models/cluecorpussmall_wwm_roberta_xlarge_seq128_model \
--world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
--total_steps 500000 --save_checkpoint_steps 50000 --report_steps 500 \
--learning_rate 2e-5 --batch_size 128 --deep_init \
--whole_word_masking --deepspeed_checkpoint_activations \
--data_processor mlm --target mlm
```
Before stage2, we extract fp32 consolidated weights from a zero 2 and 3 DeepSpeed checkpoints:
```
python3 models/cluecorpussmall_wwm_roberta_xlarge_seq128_model/zero_to_fp32.py models/cluecorpussmall_wwm_roberta_xlarge_seq128_model/ \
models/cluecorpussmall_wwm_roberta_xlarge_seq128_model.bin
```
Stage2:
```
python3 preprocess.py --corpus_path corpora/cluecorpussmall.txt \
--vocab_path models/google_zh_vocab.txt \
--dataset_path cluecorpussmall_seq512_dataset.pt \
--processes_num 32 --seq_length 512 \
--dynamic_masking --data_processor mlm
```
```
deepspeed pretrain.py --deepspeed --deepspeed_config models/deepspeed_config.json --dataset_path cluecorpussmall_seq512_dataset.pt \
--vocab_path models/google_zh_vocab.txt \
--config_path models/bert/xlarge_config.json \
--pretrained_model_path models/cluecorpussmall_wwm_roberta_xlarge_seq128_model.bin \
--output_model_path models/cluecorpussmall_wwm_roberta_xlarge_seq512_model \
--world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
--total_steps 250000 --save_checkpoint_steps 50000 --report_steps 500 \
--learning_rate 5e-5 --batch_size 32 \
--whole_word_masking --deepspeed_checkpoint_activations \
--data_processor mlm --target mlm
```
Then, we extract fp32 consolidated weights from a zero 2 and 3 DeepSpeed checkpoints:
```
python3 models/cluecorpussmall_wwm_roberta_xlarge_seq512_model/zero_to_fp32.py models/cluecorpussmall_wwm_roberta_xlarge_seq512_model/ \
models/cluecorpussmall_wwm_roberta_xlarge_seq512_model.bin
```
Finally, we convert the pre-trained model into Huggingface's format:
```
python3 scripts/convert_bert_from_tencentpretrain_to_huggingface.py --input_model_path models/cluecorpussmall_wwm_roberta_xlarge_seq512_model.bin \
--output_model_path pytorch_model.bin \
--layers_num 36 --type mlm
```
### BibTeX entry and citation info
```
@article{zhao2019uer,
title={UER: An Open-Source Toolkit for Pre-training Models},
author={Zhao, Zhe and Chen, Hui and Zhang, Jinbin and Zhao, Xin and Liu, Tao and Lu, Wei and Chen, Xi and Deng, Haotang and Ju, Qi and Du, Xiaoyong},
journal={EMNLP-IJCNLP 2019},
pages={241},
year={2019}
}
@article{zhao2023tencentpretrain,
title={TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities},
author={Zhao, Zhe and Li, Yudong and Hou, Cheng and Zhao, Jing and others},
journal={ACL 2023},
pages={217},
year={2023}
}
```
|