# What does this PR do?
This PR adds the possibility to perform GTPQ quantization with transformers model using [optimum](https://github.com/huggingface/optimum)  library. The backend relies on [auto_gptq](https://github.com/PanQiWei/AutoGPTQ) library where we use `GTPQ` and `QuantLinear` class. 
Here's the related [PR](https://github.com/huggingface/optimum/pull/1216) on optimum side. This PR can only be merged after the optimum one. 

### Quantize model 
Unlike `bitsandbytes`, it is not feasible to quantize the weights right after loading them in order to reduce memory consumption. This is why, we need to first load the entire model and then quantize it (all done in `from_pretrained`)
```py
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig
model_name = "facebook/opt-125m"

tokenizer = AutoTokenizer.from_pretrained(model_name)
config = GPTQConfig(bits=4, dataset = "c4", tokenizer=tokenizer,  group_size=128, desc_act=False)
# works also with device_map (cpu offload works but not disk offload)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, quantization_config=config)
```

### Save hf model 
We save the `quantization_config` in `model.config`
```py
# move to the device you want to save model (needed if you used device_map before)
model.to(device)
quantized_model_folder = 'opt-125m-quantized_hf'
model.save_pretrained.(save_folder)
```

### Load quantized weights
If the `model.config` has a `quantization_config` key, we will replace the layers of the model and load the quantized weights.
```py
quantized_model_from_saved = AutoModelForCausalLM.from_pretrained(quantized_model_folder,
                                                                  device_map = "auto")
```

TODO:
- [ ] merge optimum PR first
- [x] doc
GPTQ integration. # What does this PR do?
This PR adds the possibility to perform GTPQ quantization with transformers model using [optimum](https://github.com/huggingface/optimum)  library. The backend relies on [auto_gptq](https://github.com/PanQiWei/AutoGPTQ) library where we use `GTPQ` and `QuantLinear` class. 
Here's the related [PR](https://github.com/huggingface/optimum/pull/1216) on optimum side. This PR can only be merged after the optimum one. 

### Quantize model 
Unlike `bitsandbytes`, it is not feasible to quantize the weights right after loading them in order to reduce memory consumption. This is why, we need to first load the entire model and then quantize it (all done in `from_pretrained`)
```py
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig
model_name = "facebook/opt-125m"

tokenizer = AutoTokenizer.from_pretrained(model_name)
config = GPTQConfig(bits=4, dataset = "c4", tokenizer=tokenizer,  group_size=128, desc_act=False)
# works also with device_map (cpu offload works but not disk offload)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, quantization_config=config)
```

### Save hf model 
We save the `quantization_config` in `model.config`
```py
# move to the device you want to save model (needed if you used device_map before)
model.to(device)
quantized_model_folder = 'opt-125m-quantized_hf'
model.save_pretrained.(save_folder)
```

### Load quantized weights
If the `model.config` has a `quantization_config` key, we will replace the layers of the model and load the quantized weights.
```py
quantized_model_from_saved = AutoModelForCausalLM.from_pretrained(quantized_model_folder,
                                                                  device_map = "auto")
```

TODO:
- [ ] merge optimum PR first
- [x] doc
x}V{<Tyn.Eo(VXJ0IEgffshFUJ[mYiT(.UwGFy#$(̍d|3}<QtҨg''7WWK0J}GxIDw#L
80"HuǰW0k-g/kD8ޛ~ɽ	
R=p}X`0O)6VbMaNǒ	\>FSt[j^r^9HSbVcL.4vȯ%|hJ\rf*5&Fiʌ&gwE[s@B̝Fd"6ʢDKO<2O[k#X

@ &`_m{/5#jXq#g l(#Dh!pRgSI~µ;411`DC8BIR]|zQA?'aÂcdt6R|me[;B	g\H0ȓ #u6Mxcw;DP	MK6nE;M[6dAguUt[I3KMh"!!& ppc)oUekqBH"J珑s*%=mKmC=姧I;:2x6$EUM IlӤI f|%
'ks׈ׁ{=sj
q
GClNwNٰՔ>E@?"N\WOL)|djjǙz$U=@Hey+VU^d=)ۯVBl(a,?ȡ~,i{l9_r+66\VS;Wm;?ӘK]u9Rę\:lD9BTaO*=Ϙo_lpj!HJ4$uRȘxqm;%
 (0U?0LmW=[R'iI\HE":-WڡdXSjAcǾb*CzZ}qY6Ntnʹo\Sy(,`4Lb@2]js	{QY1(@<brGY,'ӽ ^a1	E|swRys'pgGڬGbC<6vm1bl~J'iʩBDP+6ɘ$2Zxu p<Q9՞aϘ.?ԏLYr4*Sx5݊GMNo.wO!zdYR:>,63[ܝL	"#?Aw-;0;Wqϝ]%kFa|Q+/OʶӿY2^GRx|eP\Me͆4v,$LsTg̙mq%</
{g7J;?iTBTBl_:N0SN+dfi&vmOCnZ^cq*_O|*W8yoRB 8ߦA;k;dZq]jlyFS奧bȁvCROLZ3?&C!(bWyO:yv]]>*g0>HltT|^O6SNag3W+8␍uOʙfj^ODl9ŒqvFCϡ0
mhU	!_Ж˨xkztx&/nXb"6ffs	ĔUEܽ~QkMzUT^^aE_f/:[z:qƩG¥+ʳ3n/+n6fIu\H!{^voܺOas{#h2VH6NRculVݦGUgwa3󖟎CxiFϯW/.'.w*%3P7X|x˺h#ЪV3kȼۣϜ"zGYǴ>7[E,B%64atь{T4?8tJ<1|h/&M-}PˈA^]FR7jGUrQ;Y2ccNk#Wan׫4
x}V{<Tyn.Eo(VXJ0IEgffshFUJ[mYiT(.UwGFy#$(̍d|3}<QtҨg''7WWK0J}GxIDw#L
80"HuǰW0k-g/kD8ޛ~ɽ	
R=p}X`0O)6VbMaNǒ	\>FSt[j^r^9HSbVcL.4vȯ%|hJ\rf*5&Fiʌ&gwE[s@B̝Fd"6ʢDKO<2O[k#X

@ &`_m{/5#jXq#g l(#Dh!pRgSI~µ;411`DC8BIR]|zQA?'aÂcdt6R|me[;B	g\H0ȓ #u6Mxcw;DP	MK6nE;M[6dAguUt[I3KMh"!!& ppc)oUekqBH"J珑s*%=mKmC=姧I;:2x6$EUM IlӤI f|%
'ks׈ׁ{=sj
q
GClNwNٰՔ>E@?"N\WOL)|djjǙz$U=@Hey+VU^d=)ۯVBl(a,?ȡ~,i{l9_r+66\VS;Wm;?ӘK]u9Rę\:lD9BTaO*=Ϙo_lpj!HJ4$uRȘxqm;%
 (0U?0LmW=[R'iI\HE":-WڡdXSjAcǾb*CzZ}qY6Ntnʹo\Sy(,`4Lb@2]js	{QY1(@<brGY,'ӽ ^a1	E|swRys'pgGڬGbC<6vm1bl~J'iʩBDP+6ɘ$2Zxu p<Q9՞aϘ.?ԏLYr4*Sx5݊GMNo.wO!zdYR:>,63[ܝL	"#?Aw-;0;Wqϝ]%kFa|Q+/OʶӿY2^GRx|eP\Me͆4v,$LsTg̙mq%</
{g7J;?iTBTBl_:N0SN+dfi&vmOCnZ^cq*_O|*W8yoRB 8ߦA;k;dZq]jlyFS奧bȁvCROLZ3?&C!(bWyO:yv]]>*g0>HltT|^O6SNag3W+8␍uOʙfj^ODl9ŒqvFCϡ0
mhU	!_Ж˨xkztx&/nXb"6ffs	ĔUEܽ~QkMzUT^^aE_f/:[z:qƩG¥+ʳ3n/+n6fIu\H!{^voܺOas{#h2VH6NRculVݦGUgwa3󖟎CxiFϯW/.'.w*%3P7X|x˺h#ЪV3kȼۣϜ"zGYǴ>7[E,B%64atь{T4?8tJ<1|h/&M-}PˈA^]FR7jGUrQ;Y2ccNk#Wan׫4