qwp4w3hyb

/

Meta-Llama-3-8B-Instruct-iMat-GGUF

like 6

I could finally test the 70B Llama-3 model with IQ3_XS quants on my MacBook with 36GB RAM. Before applying the fix, which looked like this for my case:

python3 ./llama.cpp/gguf-py/scripts/gguf-set-metadata.py Meta-Llama-3-70B-Instruct.IQ3_XS.gguf tokenizer.ggml.eos_token_id 128009 --force

Before applying the fix, the model would only stop when the max_tokens limit was reached. On my MacBook, the Q3_K_S model was just too large (but I had one that included the fix already). The IQ3_XS quantized model just worked well after increasing the RAM limit for the GPUs to 32000 with this command:

sudo sysctl iogpu.wired_limit_mb=32000