GGUF model commit (made with llama.cpp commit 0a7c980)

.gitattributes

@@ -0,0 +1,49 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+AquilaChat2-7B-16K.F16.gguf filter=lfs diff=lfs merge=lfs -text
+AquilaChat2-7B-16K.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+AquilaChat2-7B-16K.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+AquilaChat2-7B-16K.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+AquilaChat2-7B-16K.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+AquilaChat2-7B-16K.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+AquilaChat2-7B-16K.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+AquilaChat2-7B-16K.Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
+AquilaChat2-7B-16K.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+AquilaChat2-7B-16K.Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
+AquilaChat2-7B-16K.Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+AquilaChat2-7B-16K.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+AquilaChat2-7B-16K.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+AquilaChat2-7B-16K.F32.gguf filter=lfs diff=lfs merge=lfs -text

AquilaChat2-7B-16K.F16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6cc9af03b92ceb697aee8aa843d9108bbfd05c4b13eed28758bbf43db9103a42
+size 14596449856

AquilaChat2-7B-16K.F32.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c4de9d9ab0d75208768106fc8cbb4a313a1e2f20d029056c46be39577498ea6b
+size 29186991680

AquilaChat2-7B-16K.Q2_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7036d19d86d55b3e75274a0906925b5abd9263ac255d891f1e5a4ebaf985ea00
+size 2856875392

AquilaChat2-7B-16K.Q3_K_L.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9cd4d0199a6b66f17583cbf3ecc60ede36287e8271faf0c7a2d3731e51f8f461
+size 3949414016

AquilaChat2-7B-16K.Q3_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:37b04b9a620729f1a2da1d525e4a3a41fcd94cdf991e3df41def99dbb2f54533
+size 3650307712

AquilaChat2-7B-16K.Q3_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:48f9caeeb7bde29a2b8a5522a6f8642fe0b2923f1d21c64114b8ab81bc83099c
+size 3300607616

AquilaChat2-7B-16K.Q4_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ef8749c47c5ea2354196ad1d4b8d861bb4ccc2779064b8f211860f8b635cff7
+size 4215106432

AquilaChat2-7B-16K.Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:589c4c6101fa7098d922468ef1b559fdf7f7a4574e21c05b512d7641462a575f
+size 4470303616

AquilaChat2-7B-16K.Q4_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1e9118541afa9cd409b95766515576db13828166bc1708edd87fd0933b2f48d
+size 4246039424

AquilaChat2-7B-16K.Q5_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:81f58a8b6e0f295b4d290725c74ea36dd75e2b91e3b9d51c7a12538c2c317955
+size 5075811200

AquilaChat2-7B-16K.Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:efbc53a834214bb54752447b67623b44de5372622f3d018998d609f88afa9a2f
+size 5207276416

AquilaChat2-7B-16K.Q5_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49c7823abea6b31c57f7ff7b916a81dc0322e62363cd7beeceedfd894a8fd2c7
+size 5075811200

AquilaChat2-7B-16K.Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d036e986b610e8802d50fb60f6fbd6512336eef1dc22db35a8cf6bd35294f5a5
+size 5990310016

AquilaChat2-7B-16K.Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:43f00cfcdb3542ed4368b3fc46716311ca0a8c69a17a24a0fb7350af1dc93b1f
+size 7757133440

README.md

@@ -0,0 +1,309 @@
+---
+base_model: BAAI/AquilaChat2-7B-16K
+inference: false
+license: other
+model_creator: Beijing Academy of Artificial Intelligence
+model_name: Aquilachat2 7B 16K
+model_type: aquila
+prompt_template: >
+  System: A chat between a curious human and an artificial intelligence
+  assistant. The assistant gives helpful, detailed, and polite answers to the
+  human's questions.
+
+  Human: {prompt}
+
+  Assistant:
+quantized_by: mzwing
+---
+
+# AquilaChat2 7B 16K - GGUF
+- Model creator: [Beijing Academy of Artificial Intelligence](https://huggingface.co/BAAI)
+- Original model: [AquilaChat2 7B 16K](https://huggingface.co/BAAI/AquilaChat2-7B-16K)
+
+<!-- description start -->
+## Description
+
+This repo contains GGUF format model files for [Beijing Academy of Artificial Intelligence's Aquilachat2 7B 16K](https://huggingface.co/BAAI/AquilaChat2-7B-16K).
+
+These files were quantised using hardware kindly provided by [Google Colab](https://colab.research.google.com/)(Free CPU Machine).
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mzwing/AI-related/blob/master/notebooks/AquilaChat2_7B_16K_GGUF.ipynb)
+
+You can also check it out easily in [my GitHub  repo](https://github.com/mzwing/AI-related/blob/master/notebooks/AquilaChat2_7B_16K_GGUF.ipynb).
+
+<!-- description end -->
+<!-- README_GGUF.md-about-gguf start -->
+### About GGUF
+
+GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.
+
+Here is an incomplate list of clients and libraries that are known to support GGUF:
+
+* [llama.cpp](https://github.com/ggerganov/llama.cpp). The source project for GGUF. Offers a CLI and a server option.
+* [text-generation-webui](https://github.com/oobabooga/text-generation-webui), the most widely used web UI, with many features and powerful extensions. Supports GPU acceleration.
+* [KoboldCpp](https://github.com/LostRuins/koboldcpp), a fully featured web UI, with GPU accel across all platforms and GPU architectures. Especially good for story telling.
+* [LM Studio](https://lmstudio.ai/), an easy-to-use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration.
+* [LoLLMS Web UI](https://github.com/ParisNeo/lollms-webui), a great web UI with many interesting and unique features, including a full model library for easy model selection.
+* [Faraday.dev](https://faraday.dev/), an attractive and easy to use character-based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
+* [ctransformers](https://github.com/marella/ctransformers), a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.
+* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
+* [candle](https://github.com/huggingface/candle), a Rust ML framework with a focus on performance, including GPU support, and ease of use.
+* [Nitro](https://nitro.jan.ai/), a fast, lightweight 3mb inference server to supercharge apps with local AI, and OpenAI-compatible API server.
+
+<!-- README_GGUF.md-about-gguf end -->
+<!-- repositories-available start -->
+## Repositories available
+
+* [2, 3, 4, 5, 6, 8, 16 and 32-bit GGUF models for CPU+GPU inference](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF)
+* [Beijing Academy of Artificial Intelligence's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/BAAI/AquilaChat2-7B-16K)
+<!-- repositories-available end -->
+
+<!-- prompt-template start -->
+## Prompt template: AquilaChat
+
+```
+System: A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.
+Human: {prompt}
+Assistant:
+
+```
+
+<!-- prompt-template end -->
+
+<!-- compatibility_gguf start -->
+## Compatibility
+
+These quantised GGUFv2 files are compatible with llama.cpp from August 27th onwards, as of commit [d0cee0d](https://github.com/ggerganov/llama.cpp/commit/d0cee0d36d5be95a0d9088b674dbb27354107221)
+
+They are also compatible with many third party UIs and libraries - please see the list at the top of this README.
+
+## Explanation of quantisation methods
+
+<details>
+  <summary>Click to see details</summary>
+
+The new methods available are:
+
+* GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
+* GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
+* GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
+* GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
+* GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
+
+Refer to the Provided Files table below to see what files use which methods, and how.
+</details>
+<!-- compatibility_gguf end -->
+
+<!-- README_GGUF.md-provided-files start -->
+## Provided files
+
+| Name | Quant method | Bits | Size | Max RAM required | Use case |
+| ---- | ---- | ---- | ---- | ---- | ----- |
+| [AquilaChat2-7B-16K.Q2_K.gguf](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF/blob/main/AquilaChat2-7B-16K.Q2_K.gguf) | Q2_K | 2 | 2.86 GB | untested yet | smallest, significant quality loss - not recommended for most purposes |
+| [AquilaChat2-7B-16K.Q3_K_S.gguf](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF/blob/main/AquilaChat2-7B-16K.Q3_K_S.gguf) | Q3_K_S | 3 | 3.3 GB | untested yet | very small, high quality loss |
+| [AquilaChat2-7B-16K.Q3_K_M.gguf](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF/blob/main/AquilaChat2-7B-16K.Q3_K_M.gguf) | Q3_K_M | 3 | 3.65 GB | untested yet | very small, high quality loss |
+| [AquilaChat2-7B-16K.Q3_K_L.gguf](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF/blob/main/AquilaChat2-7B-16K.Q3_K_L.gguf) | Q3_K_L | 3 | 3.95 GB | untested yet | small, substantial quality loss |
+| [AquilaChat2-7B-16K.Q4_0.gguf](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF/blob/main/AquilaChat2-7B-16K.Q4_0.gguf) | Q4_0 | 4 | 4.22 GB | untested yet | legacy; small, very high quality loss - prefer using Q3_K_M |
+| [AquilaChat2-7B-16K.Q4_K_S.gguf](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF/blob/main/AquilaChat2-7B-16K.Q4_K_S.gguf) | Q4_K_S | 4 | 4.25 GB | untested yet | small, greater quality loss |
+| [AquilaChat2-7B-16K.Q4_K_M.gguf](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF/blob/main/AquilaChat2-7B-16K.Q4_K_M.gguf) | Q4_K_M | 4 | 4.47 GB | untested yet | medium, balanced quality - recommended |
+| [AquilaChat2-7B-16K.Q5_0.gguf](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF/blob/main/AquilaChat2-7B-16K.Q5_0.gguf) | Q5_0 | 5 | 5.08 GB | untested yet | legacy; medium, balanced quality - prefer using Q4_K_M |
+| [AquilaChat2-7B-16K.Q5_K_S.gguf](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF/blob/main/AquilaChat2-7B-16K.Q5_K_S.gguf) | Q5_K_S | 5 | 5.08 GB | untested yet | large, low quality loss - recommended |
+| [AquilaChat2-7B-16K.Q5_K_M.gguf](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF/blob/main/AquilaChat2-7B-16K.Q5_K_M.gguf) | Q5_K_M | 5 | 5.21 GB | untested yet | large, very low quality loss - recommended |
+| [AquilaChat2-7B-16K.Q6_K.gguf](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF/blob/main/AquilaChat2-7B-16K.Q6_K.gguf) | Q6_K | 6 | 5.99 GB | untested yet | very large, extremely low quality loss |
+| [AquilaChat2-7B-16K.Q8_0.gguf](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF/blob/main/AquilaChat2-7B-16K.Q8_0.gguf) | Q8_0 | 8 | 7.76 GB | untested yet | very large, extremely low quality loss - not recommended |
+| [AquilaChat2-7B-16K.F16.gguf](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF/blob/main/AquilaChat2-7B-16K.F16.gguf) | F16 | 16 | 14.6 GB | untested yet | extremely large, extremely low quality loss - not recommended |
+| [AquilaChat2-7B-16K.F32.gguf](https://huggingface.co/mzwing/AquilaChat2-7B-16K-GGUF/blob/main/AquilaChat2-7B-16K.F32.gguf) | F32 | 32 | 29.2 GB | untested yet | extremely large, extremely low quality loss - not recommended |
+
+**Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
+
+<!-- README_GGUF.md-provided-files end -->
+
+<!-- README_GGUF.md-how-to-download start -->
+## How to download GGUF files
+
+**Note for manual downloaders:** You almost never want to clone the entire repo! Multiple different quantisation formats are provided, and most users only want to pick and download a single file.
+
+The following clients/libraries will automatically download models for you, providing a list of available models to choose from:
+
+* LM Studio
+* LoLLMS Web UI
+* Faraday.dev
+
+### In `text-generation-webui`
+
+Under Download Model, you can enter the model repo: `mzwing/AquilaChat2-7B-16K-GGUF`, and below it, a specific filename to download, such as: `AquilaChat2-7B-16K.Q4_K_M.gguf`.
+
+Then click Download.
+
+### On the command line, including multiple files at once
+
+I recommend using the `huggingface-hub` Python library:
+
+```shell
+pip3 install huggingface-hub
+```
+
+Then you can download any individual model file to the current directory, at high speed, with a command like this:
+
+```shell
+huggingface-cli download mzwing/AquilaChat2-7B-16K-GGUF AquilaChat2-7B-16K.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
+```
+
+<details>
+  <summary>More advanced huggingface-cli download usage</summary>
+
+You can also download multiple files at once with a pattern:
+
+```shell
+huggingface-cli download mzwing/AquilaChat2-7B-16K-GGUF --local-dir . --local-dir-use-symlinks False --include='*Q4_K*gguf'
+```
+
+For more documentation on downloading with `huggingface-cli`, please see: [HF -> Hub Python Library -> Download files -> Download from the CLI](https://huggingface.co/docs/huggingface_hub/guides/download#download-from-the-cli).
+
+To accelerate downloads on fast connections (1Gbit/s or higher), install `hf_transfer`:
+
+```shell
+pip3 install hf_transfer
+```
+
+And set environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`:
+
+```shell
+HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download mzwing/AquilaChat2-7B-16K-GGUF AquilaChat2-7B-16K.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
+```
+
+Windows Command Line users: You can set the environment variable by running `set HF_HUB_ENABLE_HF_TRANSFER=1` before the download command.
+</details>
+<!-- README_GGUF.md-how-to-download end -->
+
+<!-- README_GGUF.md-how-to-run start -->
+## Example `llama.cpp` command
+
+Make sure you are using `llama.cpp` from commit [d0cee0d](https://github.com/ggerganov/llama.cpp/commit/d0cee0d36d5be95a0d9088b674dbb27354107221) or later.
+
+```shell
+./main -ngl 32 -m AquilaChat2-7B-16K.Q4_K_M.gguf --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "System: A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.\nHuman: {prompt}\nAssistant:"
+```
+
+Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
+
+Change `-c 2048` to the desired sequence length. For extended sequence models - eg 8K, 16K, 32K - the necessary RoPE scaling parameters are read from the GGUF file and set by llama.cpp automatically.
+
+If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
+
+For other parameters and how to use them, please refer to [the llama.cpp documentation](https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md)
+
+## How to run in `text-generation-webui`
+
+Further instructions here: [text-generation-webui/docs/llama.cpp.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp.md).
+
+## How to run from Python code
+
+You can use GGUF models from Python using the [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) or [ctransformers](https://github.com/marella/ctransformers) libraries.
+
+### How to load this model in Python code, using ctransformers
+
+#### First install the package
+
+Run one of the following commands, according to your system:
+
+```shell
+# Base ctransformers with no GPU acceleration
+pip install ctransformers
+# Or with CUDA GPU acceleration
+pip install ctransformers[cuda]
+# Or with AMD ROCm GPU acceleration (Linux only)
+CT_HIPBLAS=1 pip install ctransformers --no-binary ctransformers
+# Or with Metal GPU acceleration for macOS systems only
+CT_METAL=1 pip install ctransformers --no-binary ctransformers
+```
+
+#### Simple ctransformers example code
+
+```python
+from ctransformers import AutoModelForCausalLM
+
+# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
+llm = AutoModelForCausalLM.from_pretrained("mzwing/AquilaChat2-7B-16K-GGUF", model_file="AquilaChat2-7B-16K.Q4_K_M.gguf", model_type="aquila", gpu_layers=50)
+
+print(llm("AI is going to"))
+```
+
+## How to use with LangChain
+
+Here are guides on using llama-cpp-python and ctransformers with LangChain:
+
+* [LangChain + llama-cpp-python](https://python.langchain.com/docs/integrations/llms/llamacpp)
+* [LangChain + ctransformers](https://python.langchain.com/docs/integrations/providers/ctransformers)
+
+<!-- README_GGUF.md-how-to-run end -->
+
+<!-- footer start -->
+<!-- 200823 -->
+## Thanks, and how to contribute
+
+Thanks to [Google Colab](https://colab.research.google.com/)! All the quantised models in this repo are done on the awesome platform. Thanks a lot!
+
+Thanks to [llama.cpp](https://github.com/ggerganov/llama.cpp)! It inspired me to explore the inspiring AI field, thanks!
+
+Thanks to [TheBloke](https://huggingface.co/TheBloke)! Everything in this repo is a reference to him.
+
+You are welcome to create a **PullRequest**! Especially for the **RAM Usage**!
+
+<!-- footer end -->
+
+<!-- original-model-card start -->
+# Original model card: Beijing Academy of Artificial Intelligence's Aquilachat2 7B 16K
+
+
+![Aquila_logo](https://huggingface.co/BAAI/AquilaChat2-7B-16K/resolve/main/log.jpeg?download=true)
+
+
+<h4 align="center">
+    <p>
+        <b>English</b> |
+        <a href="https://huggingface.co/BAAI/AquilaChat2-7B-16K/blob/main/README_zh.md">简体中文</a> 
+    </p>
+</h4>
+
+
+We opensource our **Aquila2** series, now including **Aquila2**, the base language models, namely **Aquila2-7B** and **Aquila2-34B**, as well as **AquilaChat2**, the chat models, namely **AquilaChat2-7B** and **AquilaChat2-34B**, as well as the long-text chat models, namely **AquilaChat2-7B-16k** and **AquilaChat2-34B-16k**
+
+The additional details of the Aquila model will be presented in the official technical report. Please stay tuned for updates on official channels.
+
+## Quick Start  AquilaChat2-7B-16K（Chat model）
+
+### 1. Inference
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from transformers import BitsAndBytesConfig
+
+device = torch.device("cuda:0")
+model_info = "BAAI/AquilaChat2-7B-16K"
+tokenizer = AutoTokenizer.from_pretrained(model_info, trust_remote_code=True)
+quantization_config=BitsAndBytesConfig(
+                        load_in_4bit=True,
+                        bnb_4bit_use_double_quant=True,
+                        bnb_4bit_quant_type="nf4",
+                        bnb_4bit_compute_dtype=torch.bfloat16,
+                    )
+model = AutoModelForCausalLM.from_pretrained(model_info, trust_remote_code=True, torch_dtype=torch.float16,
+                                                # quantization_config=quantization_config, # Uncomment this line for 4bit quantization
+                                                )
+model.eval()
+model.to(device)
+text = "请给出10个要到北京旅游的理由。"
+from predict import predict
+out = predict(model, text, tokenizer=tokenizer, max_gen_len=200, top_p=0.95,
+              seed=1234, topk=100, temperature=0.9, sft=True, device=device,
+              model_name="AquilaChat2-7B-16K")
+print(out)
+```
+
+## License
+
+Aquila2 series open-source model is licensed under [BAAI Aquila Model Licence Agreement](https://huggingface.co/BAAI/AquilaChat2-7B-16K/blob/main/BAAI-Aquila-Model-License%20-Agreement.pdf)
+
+<!-- original-model-card end -->