xu-song's picture
update
751936e
raw
history blame
105 Bytes
"""
SentencePiece(unigram)
https://huggingface.co/docs/transformers/tokenizer_summary#sentencepiece
"""