Upload README.md
Browse files
README.md
CHANGED
@@ -63,7 +63,7 @@ This is a Mixtral AWQ model.
|
|
63 |
|
64 |
For AutoAWQ inference, please install AutoAWQ 0.1.8 or later.
|
65 |
|
66 |
-
Support via Transformers is
|
67 |
|
68 |
vLLM: version 0.2.6 is confirmed to support Mixtral AWQs.
|
69 |
|
@@ -113,7 +113,7 @@ Models are released as sharded safetensors files.
|
|
113 |
|
114 |
| Branch | Bits | GS | AWQ Dataset | Seq Len | Size |
|
115 |
| ------ | ---- | -- | ----------- | ------- | ---- |
|
116 |
-
| [main](https://huggingface.co/TheBloke/SauerkrautLM-Mixtral-8x7B-Instruct-AWQ/tree/main) | 4 | 128 | [
|
117 |
|
118 |
<!-- README_AWQ.md-provided-files end -->
|
119 |
|
|
|
63 |
|
64 |
For AutoAWQ inference, please install AutoAWQ 0.1.8 or later.
|
65 |
|
66 |
+
Support via Transformers is also available, but currently requires installing Transformers from Github: `pip3 install git+https://github.com/huggingface/transformers.git`
|
67 |
|
68 |
vLLM: version 0.2.6 is confirmed to support Mixtral AWQs.
|
69 |
|
|
|
113 |
|
114 |
| Branch | Bits | GS | AWQ Dataset | Seq Len | Size |
|
115 |
| ------ | ---- | -- | ----------- | ------- | ---- |
|
116 |
+
| [main](https://huggingface.co/TheBloke/SauerkrautLM-Mixtral-8x7B-Instruct-AWQ/tree/main) | 4 | 128 | [German Quad](https://huggingface.co/datasets/deepset/germanquad/viewer/) | 8192 | 24.65 GB
|
117 |
|
118 |
<!-- README_AWQ.md-provided-files end -->
|
119 |
|