Getting a ValueError upon loading the model, tensors are not matching.
#1
by
johann15
- opened
Getting this problem whenever I load it with Transformers with oobabooga. Am I missing something to load it?
"ValueError: Trying to set a tensor of shape torch.Size([1024, 5120]) in "weight" (which has shape torch.Size([1280, 5120])), this look incorrect."
You need to update transformers
run cmd_windows.bat and then you run pip install git+https://github.com/huggingface/transformers.git -U
this will update transformers and allow you to run the model.
Yes, thank you. It's working but it's pretty slow, around 2.32 t/s. I am running it on a 3090, so I guess it has to do with my config.
I get about 5t/s with load-in-8bit, might be overflowing in to ram for you
I'm uploading an exllama version in a few minutes, that gives me about 50t/s at 6.5bpw