|
--- |
|
license: mit |
|
--- |
|
|
|
Custom tokenizer used to tokenize my dataset terrycraddock/GPT2-PretrainV1-en. This tokenizer is basically the default gpt2 tokenizer except I added a [PAD] token for |
|
the trainer to perform distilled knowledge training as well as trained this on my dataset to be more efficent. |
|
|
|
Link to tokenized dataset: |
|
https://huggingface.co/datasets/terrycraddock/GPT2-PretrainV1-Tokenized-en |
|
|
|
Link to non-tokenized dataset: |
|
https://huggingface.co/datasets/terrycraddock/GPT2-PretrainV1-en |
|
|
|
This customer tokenizer as well as the datasets mentioned are intended to be used as a pretrain corpus for knowledge distillation learning from a larger GPT2 model to a |
|
smaller custom one. |
