add GPTQ-trained

README.md

@@ -21,12 +21,38 @@ tags:
 - The model responds with a structured json argument with the function name and arguments
 
 Available models:
-- fLlama-7B ([bitsandbytes NF4](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling)), ([GGML](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-GGML)), ([GPTQ](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-GPTQ)) - free
+- fLlama-7B ([bitsandbytes NF4](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling)), ([GGML](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-GGML)), ([GPTQ](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-GPTQ)), ([GPTQ-trained](https://huggingface.co/Trelis/Llama-2-7b-chat-hf-function-calling-GPTQ-trained)) - free
 - fLlama-13B ([bitsandbytes NF4](https://huggingface.co/Trelis/Llama-2-13b-chat-hf-function-calling)), ([GPTQ](https://huggingface.co/Trelis/Llama-2-13b-chat-hf-function-calling-GPTQ)) - paid
 
 ## Inference with Google Colab and HuggingFace 🤗
 
-**GPTQ (fastest + good accuracy)**
+**GPTQ-trained (fast + best accuracy) - this repo**
+All other models are from bitsandbytes NF4 training. This model is specifically trained using GPTQ methods.
+
+It is currently trickier to run because it's an adapter model. Try:
+
+```
+!pip install -q git+https://github.com/SunMarc/transformers.git@gptq_integration
+!pip install -q git+https://github.com/SunMarc/optimum.git@add-gptq-marc
+!pip install -q git+https://github.com/SunMarc/peft.git@peft_gptq
+!pip install -q git+https://github.com/fxmarty/AutoGPTQ.git@patch-act-order-exllama #probably could speed this up by using wheels. takes 5 mins right now.
+
+import transformers
+from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig, pipeline
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from auto_gptq.nn_modules.qlinear.qlinear_cuda_old import QuantLinear
+
+# Script for model loading if using adapters
+model_name_or_path = "ybelkada/llama-7b-GPTQ-test"
+
+model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map="auto") # must be auto, cannot be cpu
+
+adapter_model_name = 'Trelis/Llama-2-7b-chat-hf-function-calling-GPTQ-trained-adapters'
+
+```
+
+**GPTQ (fast + good accuracy)**
 Get started by saving your own copy of this [function calling chatbot](https://colab.research.google.com/drive/1u8x41Jx8WWtI-nzHOgqTxkS3Q_lcjaSX?usp=sharing).
 You will be able to run inference using a free Colab notebook if you select a gpu runtime. See the notebook for more details.
 
@@ -138,29 +164,28 @@ It is recommended to handle cases where:
 - There is no json object in the response
 - The response contains text in addition to the json response
 
-## Quanitization Configurations
-
-The following `bitsandbytes` quantization config was used during training:
-- load_in_8bit: False
-- load_in_4bit: True
-- llm_int8_threshold: 6.0
-- llm_int8_skip_modules: None
-- llm_int8_enable_fp32_cpu_offload: False
-- llm_int8_has_fp16_weight: False
-- bnb_4bit_quant_type: nf4
-- bnb_4bit_use_double_quant: True
-- bnb_4bit_compute_dtype: bfloat16
 
+## Training procedure
 The following `bitsandbytes` quantization config was used during training:
-- load_in_8bit: False
-- load_in_4bit: True
-- llm_int8_threshold: 6.0
-- llm_int8_skip_modules: None
-- llm_int8_enable_fp32_cpu_offload: False
-- llm_int8_has_fp16_weight: False
-- bnb_4bit_quant_type: nf4
-- bnb_4bit_use_double_quant: True
-- bnb_4bit_compute_dtype: bfloat16
+- quant_method: gptq
+- bits: 4
+- tokenizer: None
+- dataset: None
+- group_size: 128
+- damp_percent: 0.01
+- desc_act: False
+- sym: True
+- true_sequential: True
+- use_cuda_fp16: False
+- model_seqlen: None
+- block_name_to_quantize: None
+- module_name_preceding_first_block: None
+- batch_size: 1
+- pad_token_id: None
+- disable_exllama: True
+
+### Framework versions
+- PEFT 0.5.0.dev0
 
 ~