|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
--- |
|
|
|
This is GGUF model of [saucam/aqua-qwen-0.1-110B](https://huggingface.co/saucam/aqua-qwen-0.1-110B) |
|
|
|
## Usage |
|
|
|
Download the 2 files and requantize using [llama.cpp](https://github.com/ggerganov/llama.cpp). Then use the single generated file like below: |
|
|
|
``` |
|
$ ./main -m ../aqua-qwen/ggml-model-Q4_K_M.gg |
|
uf -p "<|im_start|>user\nHow are you?<|im_end|>\n<|im_start|>assistant" -n 400 -e |
|
Log start |
|
main: build = 2874 (e0f55618) |
|
main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu |
|
main: seed = 1715672499 |
|
... |
|
... |
|
sampling order: |
|
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature |
|
generate: n_ctx = 512, n_batch = 2048, n_predict = 400, n_keep = 0 |
|
|
|
|
|
,<|im_start|>user |
|
How are you?<|im_end|> |
|
<|im_start|>assistant |
|
I am an AI, I do not have feelings. How can I assist you?<|im_end|> [end of text] |
|
|
|
llama_print_timings: load time = 4065.12 ms |
|
llama_print_timings: sample time = 1.70 ms / 19 runs ( 0.09 ms per token, 11150.23 tokens per second) |
|
llama_print_timings: prompt eval time = 2898.40 ms / 12 tokens ( 241.53 ms per token, 4.14 tokens per second) |
|
llama_print_timings: eval time = 178067.55 ms / 18 runs ( 9892.64 ms per token, 0.10 tokens per second) |
|
llama_print_timings: total time = 181014.78 ms / 30 tokens |
|
Log end |
|
``` |
