error in tokenizer

#2
by kz919 - opened

File "/import/snvm-sc-scratch1/kaizhaol/miniconda3/envs/coe/lib/python3.10/site-packages/sentencepiece/init.py", line 1038, in _func
raise IndexError('piece id is out of range.')
IndexError: piece id is out of range.

DEEPNIGHT org

Hi @kz919 , thanks for bringing this to our attention.
Could you please share more information for us to replicate the error?

Thanks.

DEEPNIGHT org

We are closing this report for now but if you have any updates on it, please feel free to re-open it or reach out to us at [email protected]

deepnight-research changed discussion status to closed

Sign up or log in to comment