Update README.md
Browse files
README.md
CHANGED
@@ -38,7 +38,7 @@ The model is pre-trained by [UER-py](https://github.com/dbiir/UER-py/) on [Tence
|
|
38 |
python3 preprocess.py --corpus_path corpora/lyric.txt \
|
39 |
--vocab_path models/google_zh_vocab.txt \
|
40 |
--dataset_path lyric_dataset.pt --processes_num 32 \
|
41 |
-
--seq_length 512 --
|
42 |
```
|
43 |
|
44 |
```
|
@@ -49,10 +49,7 @@ python3 pretrain.py --dataset_path lyric_dataset.pt \
|
|
49 |
--output_model_path models/lyric_gpt2_model.bin \
|
50 |
--world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
|
51 |
--total_steps 100000 --save_checkpoint_steps 10000 --report_steps 5000 \
|
52 |
-
--learning_rate 5e-5 --batch_size 64
|
53 |
-
--embedding word_pos --remove_embedding_layernorm \
|
54 |
-
--encoder transformer --mask causal --layernorm_positioning pre \
|
55 |
-
--target lm --tie_weights
|
56 |
```
|
57 |
|
58 |
Finally, we convert the pre-trained model into Huggingface's format:
|
|
|
38 |
python3 preprocess.py --corpus_path corpora/lyric.txt \
|
39 |
--vocab_path models/google_zh_vocab.txt \
|
40 |
--dataset_path lyric_dataset.pt --processes_num 32 \
|
41 |
+
--seq_length 512 --data_processor lm
|
42 |
```
|
43 |
|
44 |
```
|
|
|
49 |
--output_model_path models/lyric_gpt2_model.bin \
|
50 |
--world_size 8 --gpu_ranks 0 1 2 3 4 5 6 7 \
|
51 |
--total_steps 100000 --save_checkpoint_steps 10000 --report_steps 5000 \
|
52 |
+
--learning_rate 5e-5 --batch_size 64
|
|
|
|
|
|
|
53 |
```
|
54 |
|
55 |
Finally, we convert the pre-trained model into Huggingface's format:
|