Can you add a 3.0 bpw exl2 quant?
#1
by
Goldkoron
- opened
Hi, my 56gb vram setup works best with 3.0bpw for Mistral Large, so I would love if you guys could add one for that.
sadly, we do not plan to add more quants, but we did provide a measurement file so you can skip the longest step and feed it back into the re-quant of exl2: https://huggingface.co/anthracite-org/magnum-v2-123b-exl2/blob/main/measurement.json
lucyknada
changed discussion status to
closed