Gibberish output from GGUF
#1
by
Olafangensan
- opened
Got a Q8 GGUF from bartowski. Use it for roleplaying in silly tavern with ChatML preset, sampler settings from here with nothing else.
Running on KoboldCPP 1.75
! un sort
;? us! you me un us! you u
! un! fe
! so! your! the! us! the! us! un! un
but us
! one!u! you! us!
us1 un! us! you! us! un! one! one! un! not! fun! we! us!
Looks like output from a broken quant. Unfortunately, not that uncommon on GGUF. I requested GGUFs from Mradermacher, hopefully they will work fine. In the meantime, you can try using Q6 quant.
UPD: Mradermacher's quants - https://huggingface.co/mradermacher/EVA-Qwen2.5-7B-v0.0-GGUF