ocisd4
/

llama-2-tokenizer-dataprep

Model card Files Files and versions Community

llama-2-tokenizer-dataprep / README.md

AlexHung29629's picture

Update README.md

96e482f over 1 year ago

|

history blame contribute delete

179 Bytes

關閉自動添加<s>，方便產生megatron-deepspeed訓練用檔案
指定pad token為<unk>，訓練時token數才會正確，以及finetune_t0.py才能正確pack_sample