|
--- |
|
license: apache-2.0 |
|
tags: |
|
- merge |
|
- mergekit |
|
- vilm/vinallama-7b-chat |
|
--- |
|
|
|
# VinaLLaMA - State-of-the-art Vietnamese LLMs |
|
|
|
 |
|
|
|
Read our [Paper](https://huggingface.co/papers/2312.11011) |
|
|
|
Prompt Format (ChatML): |
|
|
|
``` |
|
<|im_start|>system |
|
Bạn là một trợ lí AI hữu ích. Hãy trả lời người dùng một cách chính xác. |
|
<|im_end|> |
|
<|im_start|>user |
|
Hello world!<|im_end|> |
|
<|im_start|>assistant |
|
``` |
|
|
|
## Evaluation |
|
|
|
This table is copied from VBD-Llama2 with updated results from VinaLLaMA-12.5B-chat-DUS |
|
|
|
| Model | Model size | arc_vi (acc) | hellaswag_vi (acc) | mmlu_vi (acc) | truthfulqa_vi (acc) | Average | |
|
| ------------------------ | ---------- | ------------ | ------------------ | ------------- | ------------------- | ------- | |
|
| URA-LLaMA-13B | 13B | 0,3752 | 0,4830 | 0,3973 | 0,4574 | 0,4282 | |
|
| BLOOMZ-7B | 7B | 0,3205 | 0,4930 | <b>0,3975</b> | 0,4523 | 0,4158 | |
|
| PhoGPT-7B5-Instruct | 7B | 0,2470 | 0,2578 | 0,2413 | 0,4759 | 0,3055 | |
|
| SeaLLM-7B-chat | 7B | 0,3607 | 0,5112 | 0,3339 | 0,4948 | 0,4252 | |
|
| Vietcuna-7b-v3 | 7B | 0,3419 | 0,4939 | 0,3354 | 0,4807 | 0,4130 | |
|
| VinaLLaMA-2.7B-chat | 7B | 0,3273 | 0,4814 | 0,3051 | 0,4972 | 0,4028 | |
|
| VinaLLaMA-7B-chat | 7B | 0,4239 | 0,5407 | 0,3932 | 0,5251 | 0,4707 | |
|
| VBD-LLaMA2-7B-50b | 7B | 0,3222 | 0,5195 | 0,2964 | 0,4614 | 0,3999 | |
|
| VBD-LLaMA2-7B-50b-Chat | 7B | 0,3585 | 0,5207 | 0,3444 | 0,5179 | 0,4354 | |
|
| <b>VinaLLaMA-12.5B-chat-DUS</b> | 12.5B | <b>0,4325</b>| <b>0,5816</b> | 0,3875 | <b>0,5850</b> | <b>0,4967</b> | |
|
|
|
# Merging Methods |
|
|
|
This model is a merge of the following models made with [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing): |
|
* [vilm/vinallama-7b-chat](https://huggingface.co/vilm/vinallama-7b-chat) |
|
|
|
|
|
## 🧩 Configuration |
|
|
|
```yaml |
|
slices: |
|
- sources: |
|
- model: vilm/vinallama-7b-chat |
|
layer_range: [0, 16] |
|
- sources: |
|
- model: vilm/vinallama-7b-chat |
|
layer_range: [8, 16] |
|
- sources: |
|
- model: vilm/vinallama-7b-chat |
|
layer_range: [8, 16] |
|
- sources: |
|
- model: vilm/vinallama-7b-chat |
|
layer_range: [16, 24] |
|
- sources: |
|
- model: vilm/vinallama-7b-chat |
|
layer_range: [16, 24] |
|
- sources: |
|
- model: vilm/vinallama-7b-chat |
|
layer_range: [24, 28] |
|
- sources: |
|
- model: vilm/vinallama-7b-chat |
|
layer_range: [24, 28] |
|
- sources: |
|
- model: vilm/vinallama-7b-chat |
|
layer_range: [28, 32] |
|
merge_method: passthrough |
|
dtype: bfloat16 |
|
``` |
