ancient-neo / tokenizer_config.json
supermy's picture
Upload tokenizer
33cd95a
{
"add_special_token": false,
"lowercase": true,
"name_or_path": "ancient-neo-125M-finetuning/checkpoint-19500/",
"pad_token": "[PAD]",
"special_tokens_map_file": "vocab/ancient-spm-bpe/special_tokens_map.json",
"tokenizer_class": "PreTrainedTokenizerFast"
}