dvilasuero's picture
dvilasuero HF staff
Upload tokenizer
6de1a96 verified
raw
history blame
458 Bytes
{
"<pad>": 32018,
"<|Assistant|>": 32020,
"<|EOT|>": 32021,
"<|User|>": 32019,
"<|begin▁of▁sentence|>": 32013,
"<|end▁of▁sentence|>": 32014,
"<|fim▁begin|>": 32016,
"<|fim▁end|>": 32017,
"<|fim▁hole|>": 32015,
"À": 32004,
"Á": 32002,
"õ": 32000,
"ö": 32011,
"÷": 32001,
"ø": 32006,
"ù": 32010,
"ú": 32007,
"û": 32012,
"ü": 32009,
"ý": 32003,
"þ": 32008,
"ÿ": 32005
}