Update README.md

README.md

@@ -8,8 +8,9 @@ This version has then been quantized to 4-bit using [GPTQ-for-LLaMa](https://git
 
 ## Other Koala repos
 
-I have also made these other Koala repose available:
+I have also made these other Koala models available:
 * [Unquantized 13B model in HF format](https://huggingface.co/TheBloke/koala-13B-HF)
+* [GPTQ quantized 4bit 7B model in `pt` and `safetensors` formats](https://huggingface.co/TheBloke/koala-7B-4bit-128g)
 * [Unquantized 7B model in HF format](https://huggingface.co/TheBloke/koala-7B-HF)
 * [Unquantized 7B model in GGML format for llama.cpp](https://huggingface.co/TheBloke/koala-7b-ggml-unquantized)
 
@@ -18,17 +19,24 @@ I have also made these other Koala repose available:
 This GPTQ model was quantized using [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa) with the following command:
 ```
 python3 llama.py /content/koala-13B-HF c4 --wbits 4 --true-sequential --act-order --groupsize 128 --save /content/koala-13B-4bit-128g.pt
+python3 llama.py /content/koala-13B-HF c4 --wbits 4 --true-sequential --act-order --groupsize 128 --save_safetensors /content/koala-13B-4bit-128g.safetensors
 ```
 
-I created this model using the latest Triton branch of GPTQ-for-LLaMa but I believe it can be run with the CUDA branch also.
+I created this model using the latest Triton branch of GPTQ-for-LLaMa but it can be loaded with the CUDA branch also.
 
-## How to run with text-generation-webui
+## Provided files
 
-The model files provided will not load as-is with [oobaboogas text-generation-webui](https://github.com/oobabooga/text-generation-webui).
+I have provided both a `pt` and `safetensors` file. Either should work.
 
-They require the latest version of the GPTQ code.
+If both are present in the model directory for text-generation-webui I am not sure which it chooses, so you may want to place only one in the models folder.
 
-Here are the commands I used to clone GPTQ-for-LLaMa, clone text-generation-webui, and install GPTQ into the UI:
+## How to run with `text-generation-webui`
+
+GPTQ model files provided will not load as-is with [oobaboogas text-generation-webui](https://github.com/oobabooga/text-generation-webui).
+
+These model files require the latest version of the GPTQ code.
+
+Here are the commands I used to clone the Triton branch of GPTQ-for-LLaMa, clone text-generation-webui, and install GPTQ into the UI:
 ```
 git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa
 git clone https://github.com/oobabooga/text-generation-webui
@@ -36,24 +44,22 @@ mkdir -p text-generation-webui/repositories
 ln -s GPTQ-for-LLaMa text-generation-webui/repositories/GPTQ-for-LLaMa
 ```
 
-Then install this model into `text-generation-webui/models` and run text-generation-webui as follows:
+Then install this model into `text-generation-webui/models` and launch the UI as follows:
 ```
 cd text-generation-webui
-python server.py --model koala-13B-GPTQ-4bit-128g --wbits 4 --groupsize 128 --model_type Llama
+python server.py --model koala-13B-GPTQ-4bit-128g --wbits 4 --groupsize 128 --model_type Llama # add any other command line args you want
 ```
 
 The above commands assume you have installed all dependencies for GPTQ-for-LLaMa and text-generation-webui. Please see their respective repositories for further information.
 
-If you cannot use the Triton branch for any reason, I believe it should also work to use the CUDA branch instead:
+If you cannot use the Triton branch of GPTQ for any reason, you can alternatively use the CUDA branch instead:
 ```
 git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa -b cuda
+cd GPTQ-for-LLaMa
+python setup_cuda.py install
 ```
 Then link that into `text-generation-webui/repositories` as described above.
 
-## Coming soon
-
-Tomorrow I will upload a `safetensors` file as well.
-
 ## How the Koala delta weights were merged
 
 The Koala delta weights were originally merged using the following commands, producing [koala-13B-HF](https://huggingface.co/TheBloke/koala-13B-HF):