exl2?
#1
by
bdambrosio
- opened
6 -7 bit exl2 anytime soon? (I now have 72GB vram). (I assume that would beat this GPTQ in quallity, yes?)
Qwen-based models are not yet supported by exl2. When they are, I'll create the quants as it's something I'd like to test and use as well.
Turbo's added support for Qwen models now to exl2. He's uploaded some quants here:
https://huggingface.co/turboderp/Smaug-72B-exl2
I'll upload some additional quant sizes as well shortly.
Wow. Yay! thanks! I'll check his 5 bit while waiting for a 6 or 7 bit.
So far nothing has beaten LoneStriker/MixtralOrochi8x7B-8.0bpw-h8-exl2 on my adapted PubMed benchmark