metadata
license: apache-2.0
This repository contains improved Mistral-7B quantized models in GGUF format for use with llama.cpp. The models are fully compatible with the oficial llama.cpp release and can be used out=of-the-box.
The table shows a comparison between these models and the current llama.cpp quantization approach using Wikitext perplexities for a context length of 512 tokens.
The "Quantization Error" columns in the table are defined as (PPL(quantized model) - PPL(fp16))/PPL(fp16).
| Quantization | Model file | PPL(llama.cpp) | Quantization Error | PPL(new quants) | Quantization Error |
|---|---|---|---|---|---|
| Q3_K_S | mistral-7b-q3ks.gguf | 6.0692 | 6.62% | 6.0021 | 5.44% |
| Q3_K_M | mistral-7b-q3km.gguf | 5.8894 | 3.46% | 5.8489 | 2.75% |
| Q4_K_S | mistral-7b-q4ks.gguf | 5.7764 | 1.48% | 5.7349 | 0.75% |
| Q4_K_M | mistral-7b-q4km.gguf | 5.7539 | 1.08% | 5.7259 | 0.59% |
| Q5_K_S | mistral-7b-q5ks.gguf | 5.7258 | 0.59% | 5.7100 | 0.31% |
| Q4_0 | mistral-7b-q40.gguf | 5.8189 | 2.23% | 5.7924 | 1.76% |
| Q4_1 | mistral-7b-q41.gguf | 5.8244 | 2.32% | 5.7455 | 0.94% |
| Q5_0 | mistral-7b-q50.gguf | 5.7180 | 0.45% | 5.7070 | 0.26% |
| Q5_1 | mistral-7b-q51.gguf | 5.7128 | 0.36% | 5.7057 | 0.24% |
In addition, a 2-bit model is provided (mistral-7b-q2k-extra-small.gguf). It has a perplexity of 6.7099 for a context length of 512, and 5.5744 for a context of 4096.