File size: 4,370 Bytes
e802ac3 68b6109 0739198 68b6109 e802ac3 92c365a 87abc1c 92c365a c7b8f3c b6494d7 c7b8f3c 92c365a c7b8f3c 87abc1c 0d5efee c7b8f3c 92c365a c7b8f3c 87abc1c 92c365a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 |
---
license: other
library_name: transformers
pipeline_tag: text-generation
datasets:
- RyokoAI/ShareGPT52K
- Hello-SimpleAI/HC3
tags:
- koala
- ShareGPT
- llama
- gptq
inference: false
---
# Koala: A Dialogue Model for Academic Research
This repo contains the weights of the Koala 13B model produced at Berkeley. It is the result of combining the diffs from https://huggingface.co/young-geng/koala with the original Llama 13B model.
This version has then been quantized to 4-bit and 5-bit GGML for use with [llama.cpp](https://github.com/ggerganov/llama.cpp).
## My Koala repos
I have the following Koala model repositories available:
**13B models:**
* [Unquantized 13B model in HF format](https://huggingface.co/TheBloke/koala-13B-HF)
* [GPTQ quantized 4bit 13B model in `pt` and `safetensors` formats](https://huggingface.co/TheBloke/koala-13B-GPTQ-4bit-128g)
* [4bit and 5bit models in GGML format for `llama.cpp`](https://huggingface.co/TheBloke/koala-13B-GGML)
**7B models:**
* [Unquantized 7B model in HF format](https://huggingface.co/TheBloke/koala-7B-HF)
* [Unquantized 7B model in GGML format for llama.cpp](https://huggingface.co/TheBloke/koala-7b-ggml-unquantized)
* [GPTQ quantized 4bit 7B model in `pt` and `safetensors` formats](https://huggingface.co/TheBloke/koala-7B-GPTQ-4bit-128g)
* [4bit and 5bit models in GGML format for `llama.cpp`](https://huggingface.co/TheBloke/koala-7B-GGML)
## REQUIRES LATEST LLAMA.CPP (May 12th 2023 - commit b9fd7ee)!
llama.cpp recently made a breaking change to its quantisation methods.
I have re-quantised the GGML files in this repo. Therefore you will require llama.cpp compiled on May 12th or later (commit `b9fd7ee` or later) to use them.
The previous files, which will still work in older versions of llama.cpp, can be found in branch `previous_llama`.
## How to run in `llama.cpp`
I use the following command line; adjust for your tastes and needs:
```
./main -t 18 -m koala-13B-4bit-128g.GGML.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "BEGINNING OF CONVERSATION:
USER: <PROMPT GOES HERE>
GPT:"
```
Change `-t 18` to the number of physical CPU cores you have. For example if your system has 8 cores, 16 threads, use `-t 8`.
You will require 16GB or more RAM to run this model without swapping.
## How the Koala delta weights were merged
The Koala delta weights were originally merged using the following commands, producing [koala-13B-HF](https://huggingface.co/TheBloke/koala-13B-HF):
```
git clone https://github.com/young-geng/EasyLM
git clone https://huggingface.co/TheBloke/llama-13b
mkdir koala_diffs && cd koala_diffs && wget https://huggingface.co/young-geng/koala/resolve/main/koala_13b_diff_v2
cd EasyLM
PYTHON_PATH="${PWD}:$PYTHONPATH" python \
-m EasyLM.models.llama.convert_torch_to_easylm \
--checkpoint_dir=/content/llama-13b \
--output_file=/content/llama-13b-LM \
--streaming=True
PYTHON_PATH="${PWD}:$PYTHONPATH" python \
-m EasyLM.scripts.diff_checkpoint --recover_diff=True \
--load_base_checkpoint='params::/content/llama-13b-LM' \
--load_target_checkpoint='params::/content/koala_diffs/koala_13b_diff_v2' \
--output_file=/content/koala_13b.diff.weights \
--streaming=True
PYTHON_PATH="${PWD}:$PYTHONPATH" python \
-m EasyLM.models.llama.convert_easylm_to_hf --model_size=13b \
--output_dir=/content/koala-13B-HF \
--load_checkpoint='params::/content/koala_13b.diff.weights' \
--tokenizer_path=/content/llama-13b/tokenizer.model
```
## Further info
Check out the following links to learn more about the Berkeley Koala model.
* [Blog post](https://bair.berkeley.edu/blog/2023/04/03/koala/)
* [Online demo](https://koala.lmsys.org/)
* [EasyLM: training and serving framework on GitHub](https://github.com/young-geng/EasyLM)
* [Documentation for running Koala locally](https://github.com/young-geng/EasyLM/blob/main/docs/koala.md)
## License
The model weights are intended for academic research only, subject to the
[model License of LLaMA](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md),
[Terms of Use of the data generated by OpenAI](https://openai.com/policies/terms-of-use),
and [Privacy Practices of ShareGPT](https://chrome.google.com/webstore/detail/sharegpt-share-your-chatg/daiacboceoaocpibfodeljbdfacokfjb).
Any other usage of the model weights, including but not limited to commercial usage, is strictly prohibited.
|