Update README.md

README.md

@@ -1,3 +1,90 @@
 ---
 license: other
 ---
+# Koala: A Dialogue Model for Academic Research
+This repo contains the weights of the Koala 13B model produced at Berkeley. It is the result of combining the diffs from https://huggingface.co/young-geng/koala with the original Llama 13B model.
+
+This version has then been quantized to 4bit using https://github.com/qwopqwop200/GPTQ-for-LLaMa
+
+## Other Koala repos
+
+These other versions are also available:
+* [Unquantized 13B model in HF format](https://huggingface.co/TheBloke/koala-13B-HF)
+* [Unquantized 7B model in HF format](https://huggingface.co/TheBloke/koala-7B-HF)
+* [Unquantized 7B model in GGML format for llama.cpp](https://huggingface.co/TheBloke/koala-7b-ggml-unquantized)
+
+## Quantization method
+
+This GPTQ model was quantized using [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa) with the following command:
+```
+python3 llama.py /content/koala-13B-HF c4 --wbits 4 --true-sequential --act-order --groupsize 128 --save /content/koala-13B-4bit-128g.pt
+```
+
+## How to run with text-generation-webui
+
+The model files provided will not load as-is with [oobaboogas text-generation-webui](https://github.com/oobabooga/text-generation-webui).
+
+They require the latest version of the GPTQ code.
+
+Here are the commands I used to clone GPTQ-for-LLaMa, clone text-generation-webui, and install GPTQ into the UI:
+```
+git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa
+git clone https://github.com/oobabooga/text-generation-webui
+mkdir -p text-generation-webui/repositories
+ln -s GPTQ-for-LLaMa text-generation-webui/repositories/GPTQ-for-LLaMa
+```
+
+Then install this model into `text-generation-webui/models` and run text-generation-webui as follows:
+```
+cd text-generation-webui
+python server.py --model koala-13B-4bit-128g --wbits 4 --groupsize 128 --model_type Llama
+```
+
+## Coming soon
+
+Tomorrow I will upload a `safetensors` file as well.
+
+## How to merge Koala delta weights
+
+The Koala delta weights were originally merged using the following commands, producing [koala-13B-HF](https://huggingface.co/TheBloke/koala-13B-HF):
+```
+git clone https://github.com/young-geng/EasyLM
+
+git clone https://huggingface.co/TheBloke/llama-13b
+
+mkdir koala_diffs && cd koala_diffs && wget https://huggingface.co/young-geng/koala/resolve/main/koala_13b_diff_v2
+
+cd EasyLM
+
+PYTHON_PATH="${PWD}:$PYTHONPATH" python \
+-m EasyLM.models.llama.convert_torch_to_easylm \
+--checkpoint_dir=/content/llama-13b \
+--output_file=/content/llama-13b-LM \
+--streaming=True
+
+PYTHON_PATH="${PWD}:$PYTHONPATH" python \
+-m EasyLM.scripts.diff_checkpoint --recover_diff=True \
+--load_base_checkpoint='params::/content/llama-13b-LM' \
+--load_target_checkpoint='params::/content/koala_diffs/koala_13b_diff_v2' \
+--output_file=/content/koala_13b.diff.weights \
+--streaming=True
+
+PYTHON_PATH="${PWD}:$PYTHONPATH" python \
+-m EasyLM.models.llama.convert_easylm_to_hf --model_size=7b \
+--output_dir=/content/koala-13B-HF \
+--load_checkpoint='params::/content/koala_13b.diff.weights' \
+--tokenizer_path=/content/llama-13b/tokenizer.model
+```
+
+Check out the following links to learn more about the Berkeley Koala model.
+* [Blog post](https://bair.berkeley.edu/blog/2023/04/03/koala/)
+* [Online demo](https://koala.lmsys.org/)
+* [EasyLM: training and serving framework on GitHub](https://github.com/young-geng/EasyLM)
+* [Documentation for running Koala locally](https://github.com/young-geng/EasyLM/blob/main/docs/koala.md)
+
+## License
+The model weights are intended for academic research only, subject to the
+[model License of LLaMA](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md),
+[Terms of Use of the data generated by OpenAI](https://openai.com/policies/terms-of-use),
+and [Privacy Practices of ShareGPT](https://chrome.google.com/webstore/detail/sharegpt-share-your-chatg/daiacboceoaocpibfodeljbdfacokfjb).
+Any other usage of the model weights, including but not limited to commercial usage, is strictly prohibited.