Introducing AutoRound int4 algoirhtm
#14
by
wenhuach
- opened
Hello, first and foremost, I would like to express my gratitude for your exceptional work and for sharing your model with the community. We have recently applied AutoRound to your model, achieving good results . Below are the accuracies, all tested with real quantized models in the same environment , batch_size 16 and zero shot tasks.
| Metric | BF16 | INT4 |
|---|---|---|
| Avg. | 0.4504 | 0.4470 |
| mmlu | 0.5096 | 0.5053 |
| cmmlu | 0.5486 | 0.5426 |
| ceval | 0.5394 | 0.5223 |
| gsm8k | 0.2039 | 0.2176 |
Unfortunately, we are unable to upload the quantized model due to licensing constraints. Therefore, we would appreciate it if you could generate it yourself by following the recipe links, and we are here to provide assistance.