
Model created using [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) on a [GPT-2](https://huggingface.co/gpt2) model with 4-bit quantization.

You can load this model with the AutoGPTQ library, installed with the following command:

```
pip install auto-gptq
```

You can then download the model from the hub using the following code:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

model_name = "mlabonne/gpt2-GPTQ-4bit"
tokenizer = AutoTokenizer.from_pretrained(model_name)
quantize_config = BaseQuantizeConfig.from_pretrained(model_name)
model = AutoGPTQForCausalLM.from_quantized(model_name,
                                           model_basename="gptq_model-4bit-128g",
                                           device="cuda:0",
                                           use_triton=True,
                                           use_safetensors=True,
                                           quantize_config=quantize_config)
```

This model works with the traditional [Text Generation pipeline](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.TextGenerationPipeline).

Example of generation with the input text "I have a dream":

```
I have a dream. I want someone with my face, and what I have. I want to go home. I want to be alive. I want to see my children. I dream if I have the spirit, my body, my voice,
```
{"language": ["en"], "license": "apache-2.0", "tags": ["AutoGPTQ", "4bit", "GPTQ"], "pipeline_tag": "text-generation"}
transformers gpt2 text-generation AutoGPTQ 4bit GPTQ en license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us
TAGS
#transformers #gpt2 #text-generation #AutoGPTQ #4bit #GPTQ #en #license-apache-2.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 


Model created using AutoGPTQ on a GPT-2 model with 4-bit quantization.

You can load this model with the AutoGPTQ library, installed with the following command:



You can then download the model from the hub using the following code:



This model works with the traditional Text Generation pipeline.

Example of generation with the input text "I have a dream":


TAGS
#transformers #gpt2 #text-generation #AutoGPTQ #4bit #GPTQ #en #license-apache-2.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
