Does the tokenizer of this model have a network to load successfully?
#40
by
Rnake
- opened
Does the tokenizer of this model have a network to load successfully?
You would have to get the tokenizer files to the machine using it, somehow, at least once. After that, by default transformers caches the files locally. You can also copy the files by other means, then load a tokenizer by pointing at the local directory containing the files. In this respect it's not different than other tokenizers. This model uses tiktoken, in particular, if that helps.
thanks