Update README.md

README.md

@@ -1,3 +1,23 @@
 ---
 license: apache-2.0
 ---
+
+This repository contains improved Mixtral-8x7B quantized models in GGUF format for use with `llama.cpp`. The models are fully compatible with the oficial `llama.cpp` release and can be used out-of-the-box.
+
+The table shows a comparison between these models and the current `llama.cpp` quantization approach using Wikitext perplexities for a context length of 512 tokens.
+The "Quantization Error" columns in the table are defined as `(PPL(quantized model) - PPL(int8))/PPL(int8)`.
+Running the full `fp16` Mixtral8x7b model on the systems I have available takes too long, so I'm comparing against the 8-bit quantized model, where I get `PPL = 4.1049`.
+From past experience the 8-bit quantization should be basically equivalent to `fp16`. 
+
+| Quantization | Model file | PPL(llama.cpp) | Quantization Error  | PPL(new quants) | Quantization Error |
+|--:|--:|--:|--:|--:|--:|
+|Q2_K | mixtral-8x7b-q2k.gguf | 7.4660 | 81.9% | 5.0576 | 23.2% |
+|Q3_K_S | mixtral-8x7b-q3k-small.gguf | 4.4601 | 8.65% | 4.3848 | 6.82% |
+|Q3_K_M| mixtral-8x7b-q3k-medium.gguf | 4.4194 | 7.66% | 4.2884 | 4.47% |
+|Q4_K_S| mixtral-8x7b-q4k-small.gguf | 4.2523 | 3.59% | 4.1764 | 1.74% |
+|Q4_K_M| mistral-8x7b-q4k-medium.gguf | 4.2523 | 3.59% | 4.1652 | 1.47% |
+|Q5_K_S | mixtral-7b-q5k-small.gguf | 4.1395 | 0.84% | 4.1278 | 0.56% |
+|Q4_0 | mixtral-8x7b-q40.gguf | 4.2232 | 2.88% | 4.2001 | 2.32% |
+|Q4_1 | mistral-8x7b-q41.gguf | 4.2547 | 3.65% | 4.1713 | 1.62% |
+|Q5_0 | mistral-8x7b-q50.gguf | 4.1426 | 0.92% | 4.1335 | 0.70% |
+