TheBloke's picture
Try to fix tokenizer
a7a2306
raw
history blame
388 Bytes
{
"<|assistant|>": 32004,
"<|dummy10|>": 32014,
"<|dummy11|>": 32015,
"<|dummy1|>": 32005,
"<|dummy2|>": 32006,
"<|dummy3|>": 32007,
"<|dummy4|>": 32008,
"<|dummy5|>": 32009,
"<|dummy6|>": 32010,
"<|dummy7|>": 32011,
"<|dummy8|>": 32012,
"<|dummy9|>": 32013,
"<|prefix_begin|>": 32000,
"<|prefix_end|>": 32003,
"<|prompter|>": 32002,
"<|system|>": 32001
}