|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
# The Quantized Command R Model |
|
|
|
Original Base Model: `CohereForAI/c4ai-command-r-v01`.<br> |
|
Link: [https://huggingface.co/CohereForAI/c4ai-command-r-v01](https://huggingface.co/CohereForAI/c4ai-command-r-v01) |
|
|
|
## Special Notice |
|
|
|
Please note the model is quantized by utilizing the `AutoModelForCausalLM.from_pretrained` in the `transformers` package. |
|
|
|
For the model quantized by `auto-gptq` package, please check the link here: [https://huggingface.co/shuyuej/Command-R-GPTQ](https://huggingface.co/shuyuej/Command-R-GPTQ). |
|
|
|
## Quantization Configurations |
|
``` |
|
"quantization_config": { |
|
"batch_size": 1, |
|
"bits": 4, |
|
"block_name_to_quantize": null, |
|
"cache_block_outputs": true, |
|
"damp_percent": 0.1, |
|
"dataset": null, |
|
"desc_act": false, |
|
"exllama_config": { |
|
"version": 1 |
|
}, |
|
"group_size": 128, |
|
"max_input_length": null, |
|
"model_seqlen": null, |
|
"module_name_preceding_first_block": null, |
|
"modules_in_block_to_quantize": null, |
|
"pad_token_id": null, |
|
"quant_method": "gptq", |
|
"sym": true, |
|
"tokenizer": null, |
|
"true_sequential": true, |
|
"use_cuda_fp16": false, |
|
"use_exllama": true |
|
}, |
|
``` |
|
|
|
## Source Codes |
|
Source Codes: [https://github.com/vkola-lab/medpodgpt/tree/main/quantization](https://github.com/vkola-lab/medpodgpt/tree/main/quantization). |
|
|
