Did anyone managed to convert it to ggml 4bit for llama.cpp?
#6
by
wojhoiw
- opened
Did anyone managed to convert it to ggml for llama.cpp? I'm having a hard time converting it.
Tried to convert this model and gptq model and both attempt ended up with error messages, and workarounds I searched on llama.cpp repository didn't work.
Probably because I'm doing something stupid, since I've never done model converting before...
It would be great if someone could share the ggml model.
wojhoiw
changed discussion title from
Did anyone managed to convert it to ggml for llama.cpp?
to Did anyone managed to convert it to ggml 4bit for llama.cpp?
Did anyone managed to convert it to ggml for llama.cpp?
Yes. Unfiltered. 7B-GPTQ-4bit-128g-GGML
https://huggingface.co/TheBloke/vicuna-AlekseyKorshuk-7B-GPTQ-4bit-128g-GGML/blob/main/vicuna-AlekseyKorshuk-7B-GPTQ-4bit-128g.GGML.bin