---
base_model: mistralai/Mistral-7B-v0.3
language:
- en
license: apache-2.0
model_name: Mistral 7B v0.3
pipeline_tag: text-generation
tags:
- gptq
- 8-bit
inference: false
model_creator: Mistral AI
model_type: mistral
prompt_template: '{prompt}'
quantized_by: iproskurina
base_model_relation: quantized
---

![image/png](https://cdn-uploads.huggingface.co/production/uploads/629a3dbcd496c6dcdebf41cc/RME9Zljn25hQSj8-y61oo.png)


# Mistral 7B v0.3 - GPTQ
- Model creator: [Mistral AI](https://huggingface.co/mistralai)
- Original model: [Mistral 7B v0.3](https://huggingface.co/mistralai/Mistral-7B-v0.3)

The model published in this repo was quantized to 8bit using [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ).

**Quantization details**

**All quantization parameters were taken from [GPTQ paper](https://arxiv.org/abs/2210.17323).**

GPTQ calibration data consisted of 128 random 2048 token segments from the [C4 dataset](https://huggingface.co/datasets/c4).

The grouping size used for quantization is equal to 128.

## How to use this GPTQ model from Python code

### Install the necessary packages

Requires: Transformers 4.33.0 or later, Optimum 1.12.0 or later, and AutoGPTQ 0.4.2 or later.

```shell
pip3 install --upgrade transformers optimum
# If using PyTorch 2.1 + CUDA 12.x:
pip3 install --upgrade auto-gptq
# or, if using PyTorch 2.1 + CUDA 11.x:
pip3 install --upgrade auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/
```

If you are using PyTorch 2.0, you will need to install AutoGPTQ from source. Likewise if you have problems with the pre-built wheels, you should try building from source:

```shell
pip3 uninstall -y auto-gptq
git clone https://github.com/PanQiWei/AutoGPTQ
cd AutoGPTQ
git checkout v0.5.1
pip3 install .
```

### You can then use the following code

```python

from transformers import AutoTokenizer, TextGenerationPipeline,AutoModelForCausalLM
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
pretrained_model_dir = "iproskurina/Mistral-7B-v0.3-GPTQ-8bit-g128"
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_dir, use_fast=True)
model = AutoGPTQForCausalLM.from_quantized(pretrained_model_dir, device="cuda:0", model_basename="model")
pipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer)
print(pipeline("auto-gptq is")[0]["generated_text"])
```