Does the K-quant versions use i-matrix ?
#12
by
lone17
- opened
Hi, thank you for the great work. I'd like to know if i-matrix was used when producing the K-quant versions ?
Hi @lone17
You are very welcome! Yes! My script follows these steps for all models:
- Loads the model and converts it into 16bit GGUF
- Build imatrix over diverse content
- With imatrix and over 16bit GGUF I generate all the quants
I don't want to risk it, so even if imatrix is slower and might not change much in higher K-quant I still use it just in case.
lone17
changed discussion status to
closed