File size: 1,629 Bytes
42636e7 a33cb4c 42636e7 de5874d 42636e7 17d2090 42636e7 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
---
license: apache-2.0
language:
- en
---
This is GGUF model of [saucam/aqua-qwen-0.1-110B](https://huggingface.co/saucam/aqua-qwen-0.1-110B)
## Usage
Download the 2 files and merge using [llama.cpp](https://github.com/ggerganov/llama.cpp).
```
gguf-split --merge aqua-qwen-0.1-110B-Q4_K_M-00001-of-00002.gguf aqua-qwen-0.1-110B-Q4_K_M.gguf
```
Then use the single generated file like below:
```
$ ./main -m aqua-qwen-0.1-110B-Q4_K_M.gguf -p "<|im_start|>user\nHow are you?<|im_end|>\n<|im_start|>assistant" -n 400 -e
Log start
main: build = 2874 (e0f55618)
main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
main: seed = 1715672499
llama_model_loader: loaded meta data with 20 key-value pairs and 963 tensors from aqua-qwen-0.1-110B
-Q4_K_M.gguf (version GGUF V3 (latest))
...
...
sampling order:
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature
generate: n_ctx = 512, n_batch = 2048, n_predict = 400, n_keep = 0
,<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
I am an AI, I do not have feelings. How can I assist you?<|im_end|> [end of text]
llama_print_timings: load time = 4065.12 ms
llama_print_timings: sample time = 1.70 ms / 19 runs ( 0.09 ms per token, 11150.23 tokens per second)
llama_print_timings: prompt eval time = 2898.40 ms / 12 tokens ( 241.53 ms per token, 4.14 tokens per second)
llama_print_timings: eval time = 178067.55 ms / 18 runs ( 9892.64 ms per token, 0.10 tokens per second)
llama_print_timings: total time = 181014.78 ms / 30 tokens
Log end
``` |