Model not loading, even with 4-bit quantization

#65
by soumodeep-semut - opened

Tried running this with TGI, didn't work. When loading using AutoModelForCausalLM, it is giving keyerror!!

Python 3.9.16 (main, Dec 2023, 11:55:50) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> import torch
>>> model_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
>>> model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/venv/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
  File "/home/ubuntu/venv/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 1064, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/home/ubuntu/venv/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 761, in __getitem__
    raise KeyError(key)
KeyError: 'mixtral'

Hi @soumodeep-semut , pip install -U transformers should fix your issue

Sign up or log in to comment