IQ2 IMatrix GGUF
Collection
Tiny quants
•
7 items
•
Updated
•
1
IQ2-GGUF quants of sophosympatheia/Aurora-Nights-70B-v1.0
Unlike regular GGUF quants this uses important matrix similar to Quip# to keep the quant from degrading too much even at 2bpw allowing you to run larger models on less powerful machines.
NOTE: Currently you will need experimental branches of Koboldcpp or Ooba for this to work.
Regular GGUF Quants: Here
Unclear
Kooten on discord