NowaBwagel0
/

my-new-shiny-tokenizer

Model card Files Files and versions Community

my-new-shiny-tokenizer / added_tokens.json

NowaBwagel0's picture

Upload tokenizer

b1d9ada about 1 year ago

history blame contribute delete

96 Bytes

	{
	"</s>": 5,
	"<s>": 4,
	"<unk>": 3,
	"<\|im_begin\|>": 0,
	"<\|im_end\|>": 1,
	"[PAD]": 2
	}