GGUF Quants are available
#1
by
MaziyarPanahi
- opened
Hi,
Thanks for sharing this model, here are the GGUF quants if anyone needs one: https://huggingface.co/MaziyarPanahi/firefunction-v2-GGUF
@MaziyarPanahi Would love GPTQ or exl2 quants too! I am getting AttributeError: 'LlamaCppModel' object has no attribute 'model' errors with this I think because of the tokenizer not being found.
I'll do my best for the GPTQ. For the Llama models, you need the latest Llama.cpp to make it work :)
Thank you! I made an exl2 quant, but I still haven't found a way to do gptq quants on 4x3090s. Last thing I tried was an AutoGPTQ example file but that seems to make the quant but give an error at saving time.