sarashina2-7b / tokenizer_config.json

Commit History

Update tokenizer_config.json
7a3e32e
verified

kajyuuen commited on

use LlamaTokenizer
a647f24

ryo0634 commited on

Use tool tokens
f240094

kajyuuen commited on

Use FastTokenizer
065e10a

kajyuuen commited on

add tokenizer_config
14d5c52

sho-takase commited on