macadeliccc commited on
Commit
1d48e74
1 Parent(s): 38bcb59

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +19 -7
README.md CHANGED
@@ -33,9 +33,23 @@ This model is a medium-sized MoE implementation based on [cognitivecomputations/
33
 
34
  ## Quantizations
35
 
36
- **These Quants will result in unpredicted behavior. New quants are available as I have updated the model**
37
 
38
- Quatizations provided by [TheBloke](https://huggingface.co/TheBloke/laser-dolphin-mixtral-2x7b-dpo-GGUF)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
39
 
40
  ### GGUF
41
 
@@ -45,13 +59,11 @@ Quatizations provided by [TheBloke](https://huggingface.co/TheBloke/laser-dolphi
45
 
46
  *Current AWQ [Quantizations](https://huggingface.co/macadeliccc/laser-dolphin-mixtral-2x7b-dpo-AWQ)
47
 
48
- # ExLlamav2
49
-
50
- Thanks to user [bartowski](https://huggingface.co/bartowski) we now have exllamav2 quantizations in 3.5 through 8 bpw. They are available here:
51
 
52
- + [bartowski/laser-dolphin-mixtral-2x7b-dpo-exl2](https://huggingface.co/bartowski/laser-dolphin-mixtral-2x7b-dpo-exl2)
53
 
54
- His quantizations represent the first ~13B model with GQA support. Check out his repo for more information!
55
 
56
  ## HF Spaces
57
  + GGUF chat available [here](https://huggingface.co/spaces/macadeliccc/laser-dolphin-mixtral-chat-GGUF)
 
33
 
34
  ## Quantizations
35
 
36
+ ### ExLlamav2
37
 
38
+ _These are the recommended quantizations for users that are running the model on GPU_
39
+
40
+ Thanks to user [bartowski](https://huggingface.co/bartowski) we now have exllamav2 quantizations in 3.5 through 8 bpw. They are available here:
41
+
42
+ + [bartowski/laser-dolphin-mixtral-2x7b-dpo-exl2](https://huggingface.co/bartowski/laser-dolphin-mixtral-2x7b-dpo-exl2)
43
+
44
+ His quantizations represent the first ~13B model with GQA support. Check out his repo for more information!
45
+
46
+ | Branch | Bits | lm_head bits | VRAM (4k) | VRAM (16k) | VRAM (32k) | Description |
47
+ | ----- | ---- | ------- | ------ | ------ | ------ | ------------ |
48
+ | [8_0](https://huggingface.co/bartowski/laser-dolphin-mixtral-2x7b-dpo-exl2/tree/8_0) | 8.0 | 8.0 | 13.7 GB | 15.1 GB | 17.2 GB | Maximum quality that ExLlamaV2 can produce, near unquantized performance. |
49
+ | [6_5](https://huggingface.co/bartowski/laser-dolphin-mixtral-2x7b-dpo-exl2/tree/6_5) | 6.5 | 8.0 | 11.5 GB | 12.9 GB | 15.0 GB | Near unquantized performance at vastly reduced size, **recommended**. |
50
+ | [5_0](https://huggingface.co/bartowski/laser-dolphin-mixtral-2x7b-dpo-exl2/tree/5_0) | 5.0 | 6.0 | 9.3 GB | 10.7 GB | 12.8 GB | Slightly lower quality vs 6.5, great for 12gb cards with 16k context. |
51
+ | [4_25](https://huggingface.co/bartowski/laser-dolphin-mixtral-2x7b-dpo-exl2/tree/4_25) | 4.25 | 6.0 | 8.2 GB | 9.6 GB | 11.7 GB | GPTQ equivalent bits per weight. |
52
+ | [3_5](https://huggingface.co/bartowski/laser-dolphin-mixtral-2x7b-dpo-exl2/tree/3_5) | 3.5 | 6.0 | 7.0 GB | 8.4 GB | 10.5 GB | Lower quality, not recommended. |
53
 
54
  ### GGUF
55
 
 
59
 
60
  *Current AWQ [Quantizations](https://huggingface.co/macadeliccc/laser-dolphin-mixtral-2x7b-dpo-AWQ)
61
 
62
+ ### TheBloke
 
 
63
 
64
+ **These Quants will result in unpredicted behavior. New quants are available as I have updated the model**
65
 
66
+ Quatizations provided by [TheBloke](https://huggingface.co/TheBloke/laser-dolphin-mixtral-2x7b-dpo-GGUF)
67
 
68
  ## HF Spaces
69
  + GGUF chat available [here](https://huggingface.co/spaces/macadeliccc/laser-dolphin-mixtral-chat-GGUF)