Update README.md
Browse files
README.md
CHANGED
@@ -36,11 +36,16 @@ will report soon
|
|
36 |
`garage-bAInd/Platypus2-70B` was instruction fine-tuned using gptq on 2 L40 48GB.
|
37 |
|
38 |
## How to Get Started with the Model
|
|
|
|
|
|
|
|
|
39 |
|
40 |
Use the code sample provided in the original post to interact with the model.
|
41 |
```python
|
42 |
-
from transformers import
|
43 |
-
|
|
|
44 |
model_id = "malhajar/Platypus2-70B-instruct-4bit-gptq"
|
45 |
model = AutoGPTQForCausalLM.from_quantized(model_id,inject_fused_attention=False,
|
46 |
use_safetensors=True,
|
|
|
36 |
`garage-bAInd/Platypus2-70B` was instruction fine-tuned using gptq on 2 L40 48GB.
|
37 |
|
38 |
## How to Get Started with the Model
|
39 |
+
First install auto_gptq with
|
40 |
+
```shell
|
41 |
+
pip install auto_gptq
|
42 |
+
```
|
43 |
|
44 |
Use the code sample provided in the original post to interact with the model.
|
45 |
```python
|
46 |
+
from transformers import AutoTokenizer
|
47 |
+
from auto_gptq import AutoGPTQForCausalLM
|
48 |
+
|
49 |
model_id = "malhajar/Platypus2-70B-instruct-4bit-gptq"
|
50 |
model = AutoGPTQForCausalLM.from_quantized(model_id,inject_fused_attention=False,
|
51 |
use_safetensors=True,
|