christopher
/

arxiv-pyserini-tokenizer

Model card Files Files and versions Community

arxiv-pyserini-tokenizer / tokenizer_config.json

cakiki's picture

Upload tokenizer

37dae64 almost 2 years ago

history blame contribute delete

129 Bytes

	{
	"model_max_length": 1000000000000000019884624838656,
	"tokenizer_class": "PreTrainedTokenizerFast",
	"vocab_size": 70000
	}