Hugging Face's logo

bartowski
/
bigstral-12b-32k-8xMoE-GGUF

Text Generation
Transformers
GGUF
mergekit
Merge
Inference Endpoints
conversational
Model card Files Community
bartowski commited on
Commit
0dbb64a
•
1 Parent(s): a4ab7cd

Llamacpp quants

Browse files
Files changed (30) hide show
  1. .gitattributes +21 -0
  2. README.md +35 -0
  3. bigstral-12b-32k-8xMoE-Q2_K.gguf +3 -0
  4. bigstral-12b-32k-8xMoE-Q3_K_L.gguf +3 -0
  5. bigstral-12b-32k-8xMoE-Q3_K_S.gguf +3 -0
  6. bigstral-12b-32k-8xMoE-Q4_0.gguf +3 -0
  7. bigstral-12b-32k-8xMoE-Q5_0.gguf/bigstral-12b-32k-8xMoE-Q5_0_part_a +3 -0
  8. bigstral-12b-32k-8xMoE-Q5_0.gguf/bigstral-12b-32k-8xMoE-Q5_0_part_b +3 -0
  9. bigstral-12b-32k-8xMoE-Q5_0.gguf/combine.sh +2 -0
  10. bigstral-12b-32k-8xMoE-Q5_K_M.gguf/bigstral-12b-32k-8xMoE-Q5_K_M_part_a +3 -0
  11. bigstral-12b-32k-8xMoE-Q5_K_M.gguf/bigstral-12b-32k-8xMoE-Q5_K_M_part_b +3 -0
  12. bigstral-12b-32k-8xMoE-Q5_K_M.gguf/combine.sh +2 -0
  13. bigstral-12b-32k-8xMoE-Q5_K_S.gguf/bigstral-12b-32k-8xMoE-Q5_K_S_part_a +3 -0
  14. bigstral-12b-32k-8xMoE-Q5_K_S.gguf/bigstral-12b-32k-8xMoE-Q5_K_S_part_b +3 -0
  15. bigstral-12b-32k-8xMoE-Q5_K_S.gguf/combine.sh +2 -0
  16. bigstral-12b-32k-8xMoE-Q6_K.gguf/bigstral-12b-32k-8xMoE-Q6_K_part_a +3 -0
  17. bigstral-12b-32k-8xMoE-Q6_K.gguf/bigstral-12b-32k-8xMoE-Q6_K_part_b +3 -0
  18. bigstral-12b-32k-8xMoE-Q6_K.gguf/combine.sh +1 -0
  19. bigstral-12b-32k-8xMoE-Q8_0.gguf/bigstral-12b-32k-8xMoE-Q8_0_part_a +3 -0
  20. bigstral-12b-32k-8xMoE-Q8_0.gguf/bigstral-12b-32k-8xMoE-Q8_0_part_b +3 -0
  21. bigstral-12b-32k-8xMoE-Q8_0.gguf/bigstral-12b-32k-8xMoE-Q8_0_part_c +3 -0
  22. bigstral-12b-32k-8xMoE-Q8_0.gguf/bigstral-12b-32k-8xMoE-Q8_0_part_d +1 -0
  23. bigstral-12b-32k-8xMoE-Q8_0.gguf/combine.sh +2 -0
  24. bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_a +3 -0
  25. bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_b +3 -0
  26. bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_c +3 -0
  27. bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_d +3 -0
  28. bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_e +3 -0
  29. bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_f +3 -0
  30. bigstral-12b-32k-8xMoE-fp16.gguf/combine.sh +2 -0
.gitattributes CHANGED
@@ -36,3 +36,24 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
36
  bigstral-12b-32k-8xMoE-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
37
  bigstral-12b-32k-8xMoE-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
38
  bigstral-12b-32k-8xMoE-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
  bigstral-12b-32k-8xMoE-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
37
  bigstral-12b-32k-8xMoE-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
38
  bigstral-12b-32k-8xMoE-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
39
+ bigstral-12b-32k-8xMoE-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
40
+ bigstral-12b-32k-8xMoE-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
41
+ bigstral-12b-32k-8xMoE-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
42
+ bigstral-12b-32k-8xMoE-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
43
+ bigstral-12b-32k-8xMoE-Q5_0.gguf/bigstral-12b-32k-8xMoE-Q5_0_part_a filter=lfs diff=lfs merge=lfs -text
44
+ bigstral-12b-32k-8xMoE-Q5_0.gguf/bigstral-12b-32k-8xMoE-Q5_0_part_b filter=lfs diff=lfs merge=lfs -text
45
+ bigstral-12b-32k-8xMoE-Q5_K_M.gguf/bigstral-12b-32k-8xMoE-Q5_K_M_part_a filter=lfs diff=lfs merge=lfs -text
46
+ bigstral-12b-32k-8xMoE-Q5_K_M.gguf/bigstral-12b-32k-8xMoE-Q5_K_M_part_b filter=lfs diff=lfs merge=lfs -text
47
+ bigstral-12b-32k-8xMoE-Q5_K_S.gguf/bigstral-12b-32k-8xMoE-Q5_K_S_part_a filter=lfs diff=lfs merge=lfs -text
48
+ bigstral-12b-32k-8xMoE-Q5_K_S.gguf/bigstral-12b-32k-8xMoE-Q5_K_S_part_b filter=lfs diff=lfs merge=lfs -text
49
+ bigstral-12b-32k-8xMoE-Q6_K.gguf/bigstral-12b-32k-8xMoE-Q6_K_part_a filter=lfs diff=lfs merge=lfs -text
50
+ bigstral-12b-32k-8xMoE-Q6_K.gguf/bigstral-12b-32k-8xMoE-Q6_K_part_b filter=lfs diff=lfs merge=lfs -text
51
+ bigstral-12b-32k-8xMoE-Q8_0.gguf/bigstral-12b-32k-8xMoE-Q8_0_part_a filter=lfs diff=lfs merge=lfs -text
52
+ bigstral-12b-32k-8xMoE-Q8_0.gguf/bigstral-12b-32k-8xMoE-Q8_0_part_b filter=lfs diff=lfs merge=lfs -text
53
+ bigstral-12b-32k-8xMoE-Q8_0.gguf/bigstral-12b-32k-8xMoE-Q8_0_part_c filter=lfs diff=lfs merge=lfs -text
54
+ bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_a filter=lfs diff=lfs merge=lfs -text
55
+ bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_b filter=lfs diff=lfs merge=lfs -text
56
+ bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_c filter=lfs diff=lfs merge=lfs -text
57
+ bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_d filter=lfs diff=lfs merge=lfs -text
58
+ bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_e filter=lfs diff=lfs merge=lfs -text
59
+ bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_f filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model:
3
+ - mistralai/Mistral-7B-Instruct-v0.2
4
+ library_name: transformers
5
+ tags:
6
+ - mergekit
7
+ - merge
8
+ quantized_by: bartowski
9
+ pipeline_tag: text-generation
10
+ ---
11
+
12
+ ## Llamacpp Quantizations of bigstral-12b-32k-8xMoE
13
+
14
+ Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b2354">b2354</a> for quantization.
15
+
16
+ Original model: https://huggingface.co/bartowski/bigstral-12b-32k-8xMoE
17
+
18
+ Download a file (not the whole branch) from below:
19
+
20
+ | Filename | Quant type | File Size | Description |
21
+ | -------- | ---------- | --------- | ----------- |
22
+ | [bigstral-12b-32k-8xMoE-Q8_0.gguf](https://huggingface.co/bartowski/bigstral-12b-32k-8xMoE-GGUF/blob/main/bigstral-12b-32k-8xMoE-Q8_0.gguf) | Q8_0 | 86.63GB | Extremely high quality, generally unneeded but max available quant. |
23
+ | [bigstral-12b-32k-8xMoE-Q6_K.gguf](https://huggingface.co/bartowski/bigstral-12b-32k-8xMoE-GGUF/blob/main/bigstral-12b-32k-8xMoE-Q6_K.gguf) | Q6_K | 67.00GB | Very high quality, near perfect, *recommended*. |
24
+ | [bigstral-12b-32k-8xMoE-Q5_K_M.gguf](https://huggingface.co/bartowski/bigstral-12b-32k-8xMoE-GGUF/blob/main/bigstral-12b-32k-8xMoE-Q5_K_M.gguf) | Q5_K_M | 58.00GB | High quality, very usable. |
25
+ | [bigstral-12b-32k-8xMoE-Q5_K_S.gguf](https://huggingface.co/bartowski/bigstral-12b-32k-8xMoE-GGUF/blob/main/bigstral-12b-32k-8xMoE-Q5_K_S.gguf) | Q5_K_S | 56.25GB | High quality, very usable. |
26
+ | [bigstral-12b-32k-8xMoE-Q5_0.gguf](https://huggingface.co/bartowski/bigstral-12b-32k-8xMoE-GGUF/blob/main/bigstral-12b-32k-8xMoE-Q5_0.gguf) | Q5_0 | 56.25GB | High quality, older format, generally not recommended. |
27
+ | [bigstral-12b-32k-8xMoE-Q4_K_M.gguf](https://huggingface.co/bartowski/bigstral-12b-32k-8xMoE-GGUF/blob/main/bigstral-12b-32k-8xMoE-Q4_K_M.gguf) | Q4_K_M | 49.60GB | Good quality, similar to 4.25 bpw. |
28
+ | [bigstral-12b-32k-8xMoE-Q4_K_S.gguf](https://huggingface.co/bartowski/bigstral-12b-32k-8xMoE-GGUF/blob/main/bigstral-12b-32k-8xMoE-Q4_K_S.gguf) | Q4_K_S | 46.70GB | Slightly lower quality with small space savings. |
29
+ | [bigstral-12b-32k-8xMoE-Q4_0.gguf](https://huggingface.co/bartowski/bigstral-12b-32k-8xMoE-GGUF/blob/main/bigstral-12b-32k-8xMoE-Q4_0.gguf) | Q4_0 | 46.13GB | Decent quality, older format, generally not recommended. |
30
+ | [bigstral-12b-32k-8xMoE-Q3_K_L.gguf](https://huggingface.co/bartowski/bigstral-12b-32k-8xMoE-GGUF/blob/main/bigstral-12b-32k-8xMoE-Q3_K_L.gguf) | Q3_K_L | 42.16GB | Lower quality but usable, good for low RAM availability. |
31
+ | [bigstral-12b-32k-8xMoE-Q3_K_M.gguf](https://huggingface.co/bartowski/bigstral-12b-32k-8xMoE-GGUF/blob/main/bigstral-12b-32k-8xMoE-Q3_K_M.gguf) | Q3_K_M | 39.30GB | Even lower quality. |
32
+ | [bigstral-12b-32k-8xMoE-Q3_K_S.gguf](https://huggingface.co/bartowski/bigstral-12b-32k-8xMoE-GGUF/blob/main/bigstral-12b-32k-8xMoE-Q3_K_S.gguf) | Q3_K_S | 35.62GB | Low quality, not recommended. |
33
+ | [bigstral-12b-32k-8xMoE-Q2_K.gguf](https://huggingface.co/bartowski/bigstral-12b-32k-8xMoE-GGUF/blob/main/bigstral-12b-32k-8xMoE-Q2_K.gguf) | Q2_K | 30.17GB | Extremely low quality, *not* recommended.
34
+
35
+ Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski
bigstral-12b-32k-8xMoE-Q2_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e252fc34fd5d88aeb6441fbfaebaf5220103263afa7b35c69a7ddf99b98c29bc
3
+ size 30177607008
bigstral-12b-32k-8xMoE-Q3_K_L.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:24531c929491ee0c5fc336f5fa6b02babcb1bef38a3ccb027d694266b8650007
3
+ size 42169851232
bigstral-12b-32k-8xMoE-Q3_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:244d763db6ffb0c1b7f6ea992de64c872dc8ea572feeb6cd91e6ccae0537fe91
3
+ size 35629882720
bigstral-12b-32k-8xMoE-Q4_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6be986a45be187b34ada119715ccf683e0f0dbb6476664cd66845f96403fcd8f
3
+ size 46136196448
bigstral-12b-32k-8xMoE-Q5_0.gguf/bigstral-12b-32k-8xMoE-Q5_0_part_a ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c64c0a3c094cde2b564c09e52ed9aea10ac60b173a36876ddf3f496791260ae1
3
+ size 28126232240
bigstral-12b-32k-8xMoE-Q5_0.gguf/bigstral-12b-32k-8xMoE-Q5_0_part_b ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bb2dcfe4ce4a6e1b1cb2233155b39e7d7c7fcfa8b60fb0b1933621d381c27bf8
3
+ size 28126232240
bigstral-12b-32k-8xMoE-Q5_0.gguf/combine.sh ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ #!/bin/bash
2
+ cat bigstral-12b-32k-8xMoE-Q5_0_part_* > "bigstral-12b-32k-8xMoE-Q5_0.gguf"
bigstral-12b-32k-8xMoE-Q5_K_M.gguf/bigstral-12b-32k-8xMoE-Q5_K_M_part_a ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:91a6313134b1f98a254d04e5afdc80d61c1054d70a45a2afa69997cc5db140b5
3
+ size 28999696048
bigstral-12b-32k-8xMoE-Q5_K_M.gguf/bigstral-12b-32k-8xMoE-Q5_K_M_part_b ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a0bbfdc9d545e98634e0489a38399dcefd508be0a00b7a0e3106aa5b9d30654d
3
+ size 28999696048
bigstral-12b-32k-8xMoE-Q5_K_M.gguf/combine.sh ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ #!/bin/bash
2
+ cat bigstral-12b-32k-8xMoE-Q5_K_M_part_* > "bigstral-12b-32k-8xMoE-Q5_K_M.gguf"
bigstral-12b-32k-8xMoE-Q5_K_S.gguf/bigstral-12b-32k-8xMoE-Q5_K_S_part_a ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b0fe22f1f8b3af151d9e838421fced4b0dfe947b92f422965ae6a63afe351e50
3
+ size 28126232240
bigstral-12b-32k-8xMoE-Q5_K_S.gguf/bigstral-12b-32k-8xMoE-Q5_K_S_part_b ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5052e23f28bfb3d598ee9350dd69e67931dee0f91e79bcea8bb668ee80d8abcc
3
+ size 28126232240
bigstral-12b-32k-8xMoE-Q5_K_S.gguf/combine.sh ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ #!/bin/bash
2
+ cat bigstral-12b-32k-8xMoE-Q5_K_S_part_* > "bigstral-12b-32k-8xMoE-Q5_K_S.gguf"
bigstral-12b-32k-8xMoE-Q6_K.gguf/bigstral-12b-32k-8xMoE-Q6_K_part_a ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3fdd3cb57b17ab979c4a4c4c815cdd17a11faaa29916f50fe06a3f9d3fd32a11
3
+ size 33500499632
bigstral-12b-32k-8xMoE-Q6_K.gguf/bigstral-12b-32k-8xMoE-Q6_K_part_b ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7f6ecef017cfe4e6febd366ee4dcfd0645fb7a27625952967097b3c78ac78d75
3
+ size 33500499632
bigstral-12b-32k-8xMoE-Q6_K.gguf/combine.sh ADDED
@@ -0,0 +1 @@
 
 
1
+ cat bigstral-12b-32k-8xMoE-Q6_K_part_* > bigstral-12b-32k-8xMoE-Q6_K.gguf
bigstral-12b-32k-8xMoE-Q8_0.gguf/bigstral-12b-32k-8xMoE-Q8_0_part_a ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5ec0c57cf962977b6d8d96b9138a83d4b56b526e92497caab9176b44d1882596
3
+ size 28877670858
bigstral-12b-32k-8xMoE-Q8_0.gguf/bigstral-12b-32k-8xMoE-Q8_0_part_b ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7114908d71d2da3acf9c1877483b2cd574a8268da145660a106887dd80b1ee7d
3
+ size 28877670858
bigstral-12b-32k-8xMoE-Q8_0.gguf/bigstral-12b-32k-8xMoE-Q8_0_part_c ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:62e7148ad969f5c22ef9a1c6f2239fb6d9f9f11f3076d7f1664be812f986ce02
3
+ size 28877670858
bigstral-12b-32k-8xMoE-Q8_0.gguf/bigstral-12b-32k-8xMoE-Q8_0_part_d ADDED
@@ -0,0 +1 @@
 
 
1
+ 
bigstral-12b-32k-8xMoE-Q8_0.gguf/combine.sh ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ #!/bin/bash
2
+ cat bigstral-12b-32k-8xMoE-Q8_0_part_* > "bigstral-12b-32k-8xMoE-Q8_0.gguf"
bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_a ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:aa2b2336a0888e4daf61944398b923ae7e68f68606e6cc70b110cfc19bad648e
3
+ size 27178050784
bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_b ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1086e3256475867269966433255942480971345192c7c31dd81fbc7a3ac0b683
3
+ size 27178050784
bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_c ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0b76c4b4f8720cf721099b9667eefbf2e53c871dbe670d066f591b19e6fadf20
3
+ size 27178050784
bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_d ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e5598bf5e2d82c2a040891d1f1130b95c23f595b4308154b249a698810365d22
3
+ size 27178050784
bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_e ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:29db9161be9eb598018ba2563ec693b4876e00f9dff25103cfa431babb2b5737
3
+ size 27178050784
bigstral-12b-32k-8xMoE-fp16.gguf/bigstral-12b-32k-8xMoE-fp16_part_f ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:721ec2fa1882339cfaa5987c32dd16f3f7f92271fcbc395d748180f0296f2c98
3
+ size 27178050784
bigstral-12b-32k-8xMoE-fp16.gguf/combine.sh ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ #!/bin/bash
2
+ cat bigstral-12b-32k-8xMoE-fp16_part_* > "bigstral-12b-32k-8xMoE-fp16.gguf"