loubnabnl
/

santacoder-code-to-text

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

santacoder-code-to-text / tokenizer_config.json

loubnabnl's picture

loubnabnl HF staff

update tokenizer

021e008 almost 2 years ago

363 Bytes

	{
	"name_or_path": "bigcode/digit-bytelevel-bpe-jss-v1.1-49152",
	"special_tokens_map_file": "/Users/leandro/.cache/huggingface/hub/models--bigcode--digit-bytelevel-bpe-jss-v1.1-49152/snapshots/fa09b77949689a484afafc5f89534e6b6ba2c151/special_tokens_map.json",
	"tokenizer_class": "PreTrainedTokenizerFast",
	"vocab_size": 49152,
	"model_max_length": 2048
	}