Or any other quants to fit in a 3090/4090?
I think the amazing @TheBloke is going to convert them and push them as always to his repos!You can already use load_in_4bit = True for bitsandbytes quantization with transformers format
load_in_4bit = True
bitsandbytes
· Sign up or log in to comment