weights does not exist when trying to deploy in sagemaker endpoint
#1
by
LorenzoCevolaniAXA
- opened
I am trying to run this model inside a sagemaker endpoint.
I have received the following error:
'''
weight model.layers.0.self_attn.q_proj.weight does not exist
'''
is it something expected? do we need to do something more than downloading and loading the weights?
Thanks
This checkpoint is currently meant for running in vLLM https://huggingface.co/neuralmagic/Meta-Llama-3.1-405B-Instruct-quantized.w4a16#deployment
It will be able to run in Transformers once this PR lands! https://github.com/huggingface/transformers/pull/31704