jeiku

/

Neo-e2

like 1

I am struggling with these new models.

getting this error when attempting to quantize for AQW, using the master branch of AutoAWQ.

Starting quantization for model at /opt/openbet/inference/data/jeiku-Neo-e2
Using CUDA. Available GPU memory: 23.60 GB
Loading checkpoint shards:   0%|                                                                                            | 0/2 [00:00<?, ?it/s]
Quantization failed: Trying to set a tensor of shape torch.Size([1024, 3072]) in "weight" (which has shape torch.Size([768, 3072])), this looks incorrect.
Quantization failed: Trying to set a tensor of shape torch.Size([1024, 3072]) in "weight" (which has shape torch.Size([768, 3072])), this looks incorrect.
An error occurred during the quantization process: Trying to set a tensor of shape torch.Size([1024, 3072]) in "weight" (which has shape torch.Size([768, 3072])), this looks incorrect.