TheBloke

/

Llama-2-70B-Chat-GPTQ

like 260

The model is giving me a bunch of nonsense output "íliaa enemies enemiesclicclic Sue SueUob pil Silvererde wc Sept", and if I ran it again it broke
'RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.'

  model = AutoGPTQForCausalLM.from_quantized(model_name, 
  model_basename="gptq_model-4bit--1g",
  device_map="auto", 
  use_safetensors=True, 
  use_triton=False, 
  trust_remote_code=True, 
  quantize_config=None,
  inject_fused_attention=False)