BelleGroup

/

BELLE-LLaMA-13B-2M-enc

like 24

Follow

BELLE Group // Be Everyone's Large Language model Engine 182

ValueError: Couldn't instantiate the backend tokenizer from one of:
(1) a tokenizers library serialization file,
(2) a slow tokenizer instance to convert or
(3) an equivalent slow tokenizer class to instantiate and convert.
You need to have sentencepiece installed to convert a slow tokenizer to a fast one.

已经安装了sentencepiece，但是还是报错了。
尝试使用LlamaTokenizer代替AutoTokenizer则运行成功了。
请问这样的解决方案合理么，问题出在哪里？