Qwen/Qwen2-7B-Instruct · load from local filepath error

Jun 6

•

The model loads fine, but when I try to load the tokenizer: transformers.from_pretrained( 'local filepath')

File "/home/bruce/.local/lib/python3.10/site-packages/transformers/models/qwen2/tokenization_qwen2.py", line 179, in init
with open(merges_file, encoding="utf-8") as merges_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType

btw, using transformers 4.41.1

jklj077

Qwen org Jun 7

I'm not sure what is transformers.from_pretrained( 'local filepath'), but the following should be fine:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("local filepath/")

Byerose

11 days ago

same question.
I tried:

model_id = "Qwen2.5-72B-Instruct"
model_path = os.path.join("/disk/mount/models/", model_id)
model_path += "/"
tokenizer = AutoTokenizer.from_pretrained(model_path)

don't work...

  File "main.py", line 90, in <module>
    tokenizer = AutoTokenizer.from_pretrained(model_path)  # for qwen
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 926, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2208, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2246, in _from_pretrained
    slow_tokenizer = (cls.slow_tokenizer_class)._from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2442, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.11/site-packages/transformers/models/qwen2/tokenization_qwen2.py", line 172, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected str, bytes or os.PathLike object, not NoneType