How to use the model in Windows?
#3
by
kristada673
- opened
I copy-pasted your Python script and ran it, gives the following error:
C:\Users\Username\Documents>python script.py
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ C:\Users\Username\Documents\script.py:12 in <module> │
│ │
│ 9 │
│ 10 tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True) │
│ 11 │
│ ❱ 12 model = AutoGPTQForCausalLM.from_quantized(model_name_or_path, │
│ 13 │ │ model_basename=model_basename, │
│ 14 │ │ use_safetensors=True, │
│ 15 │ │ trust_remote_code=True, │
│ │
│ C:\Users\Username\anaconda3\lib\site-packages\auto_gptq\modeling\auto.py:63 in from_quantized │
│ │
│ 60 │ │ trust_remote_code: bool = False │
│ 61 │ ) -> BaseGPTQForCausalLM: │
│ 62 │ │ model_type = check_and_get_model_type(save_dir) │
│ ❱ 63 │ │ return GPTQ_CAUSAL_LM_MODEL_MAP[model_type].from_quantized( │
│ 64 │ │ │ save_dir=save_dir, │
│ 65 │ │ │ device=device, │
│ 66 │ │ │ use_safetensors=use_safetensors, │
│ │
│ C:\Users\Username\anaconda3\lib\site-packages\auto_gptq\modeling\_base.py:501 in │
│ from_quantized │
│ │
│ 498 │ │ │ raise TypeError(f"{config.model_type} isn't supported yet.") │
│ 499 │ │ │
│ 500 │ │ if quantize_config is None: │
│ ❱ 501 │ │ │ quantize_config = BaseQuantizeConfig.from_pretrained(save_dir) │
│ 502 │ │ │
│ 503 │ │ if model_basename is None: │
│ 504 │ │ │ model_basename = f"gptq_model-{quantize_config.bits}bit-{quantize_config.gro │
│ │
│ C:\Users\Username\anaconda3\lib\site-packages\auto_gptq\modeling\_base.py:51 in │
│ from_pretrained │
│ │
│ 48 │ │
│ 49 │ @classmethod │
│ 50 │ def from_pretrained(cls, save_dir: str): │
│ ❱ 51 │ │ with open(join(save_dir, "quantize_config.json"), "r", encoding="utf-8") as f: │
│ 52 │ │ │ return cls(**json.load(f)) │
│ 53 │ │
│ 54 │ def to_dict(self): │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [Errno 2] No such file or directory: 'TheBloke/Nous-Hermes-13B-GPTQ\\quantize_config.json'
I suspect it might be due to the directory naming structure in Windows vs Linux. But I don't want to edit the library files before ensuring that that's indeed the issue.