RichardErkhov
commited on
Commit
•
31b3813
1
Parent(s):
09803a0
uploaded readme
Browse files
README.md
ADDED
@@ -0,0 +1,410 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
Quantization made by Richard Erkhov.
|
2 |
+
|
3 |
+
[Github](https://github.com/RichardErkhov)
|
4 |
+
|
5 |
+
[Discord](https://discord.gg/pvy7H8DZMG)
|
6 |
+
|
7 |
+
[Request more models](https://github.com/RichardErkhov/quant_request)
|
8 |
+
|
9 |
+
|
10 |
+
SeaLLM-7B-v2 - GGUF
|
11 |
+
- Model creator: https://huggingface.co/SeaLLMs/
|
12 |
+
- Original model: https://huggingface.co/SeaLLMs/SeaLLM-7B-v2/
|
13 |
+
|
14 |
+
|
15 |
+
| Name | Quant method | Size |
|
16 |
+
| ---- | ---- | ---- |
|
17 |
+
| [SeaLLM-7B-v2.Q2_K.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q2_K.gguf) | Q2_K | 2.6GB |
|
18 |
+
| [SeaLLM-7B-v2.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.IQ3_XS.gguf) | IQ3_XS | 2.89GB |
|
19 |
+
| [SeaLLM-7B-v2.IQ3_S.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.IQ3_S.gguf) | IQ3_S | 3.04GB |
|
20 |
+
| [SeaLLM-7B-v2.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q3_K_S.gguf) | Q3_K_S | 3.03GB |
|
21 |
+
| [SeaLLM-7B-v2.IQ3_M.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.IQ3_M.gguf) | IQ3_M | 3.14GB |
|
22 |
+
| [SeaLLM-7B-v2.Q3_K.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q3_K.gguf) | Q3_K | 3.36GB |
|
23 |
+
| [SeaLLM-7B-v2.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q3_K_M.gguf) | Q3_K_M | 3.36GB |
|
24 |
+
| [SeaLLM-7B-v2.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q3_K_L.gguf) | Q3_K_L | 3.64GB |
|
25 |
+
| [SeaLLM-7B-v2.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.IQ4_XS.gguf) | IQ4_XS | 3.76GB |
|
26 |
+
| [SeaLLM-7B-v2.Q4_0.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q4_0.gguf) | Q4_0 | 3.91GB |
|
27 |
+
| [SeaLLM-7B-v2.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.IQ4_NL.gguf) | IQ4_NL | 3.96GB |
|
28 |
+
| [SeaLLM-7B-v2.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q4_K_S.gguf) | Q4_K_S | 3.94GB |
|
29 |
+
| [SeaLLM-7B-v2.Q4_K.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q4_K.gguf) | Q4_K | 4.16GB |
|
30 |
+
| [SeaLLM-7B-v2.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q4_K_M.gguf) | Q4_K_M | 4.16GB |
|
31 |
+
| [SeaLLM-7B-v2.Q4_1.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q4_1.gguf) | Q4_1 | 4.33GB |
|
32 |
+
| [SeaLLM-7B-v2.Q5_0.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q5_0.gguf) | Q5_0 | 4.75GB |
|
33 |
+
| [SeaLLM-7B-v2.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q5_K_S.gguf) | Q5_K_S | 4.75GB |
|
34 |
+
| [SeaLLM-7B-v2.Q5_K.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q5_K.gguf) | Q5_K | 4.87GB |
|
35 |
+
| [SeaLLM-7B-v2.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q5_K_M.gguf) | Q5_K_M | 4.87GB |
|
36 |
+
| [SeaLLM-7B-v2.Q5_1.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q5_1.gguf) | Q5_1 | 5.17GB |
|
37 |
+
| [SeaLLM-7B-v2.Q6_K.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q6_K.gguf) | Q6_K | 5.64GB |
|
38 |
+
|
39 |
+
|
40 |
+
|
41 |
+
|
42 |
+
Original model description:
|
43 |
+
---
|
44 |
+
license: other
|
45 |
+
license_name: seallms
|
46 |
+
license_link: https://huggingface.co/SeaLLMs/SeaLLM-13B-Chat/blob/main/LICENSE
|
47 |
+
language:
|
48 |
+
- en
|
49 |
+
- zh
|
50 |
+
- vi
|
51 |
+
- id
|
52 |
+
- th
|
53 |
+
- ms
|
54 |
+
- km
|
55 |
+
- lo
|
56 |
+
- my
|
57 |
+
- tl
|
58 |
+
tags:
|
59 |
+
- multilingual
|
60 |
+
- sea
|
61 |
+
---
|
62 |
+
|
63 |
+
<p align="center">
|
64 |
+
<img src="seal_logo.png" width="200" />
|
65 |
+
</p>
|
66 |
+
|
67 |
+
# *SeaLLM-7B-v2* - Large Language Models for Southeast Asia
|
68 |
+
|
69 |
+
# <strong style="color: red">BIG NEWS: <a href="https://huggingface.co/SeaLLMs/SeaLLM-7B-v2.5">SeaLLM-7B-v2.5</a> is released with state-of-the-art performance in world knowledge and reasoning. SeaLLM-7B-v2 will begin deprecation.</strong>
|
70 |
+
|
71 |
+
|
72 |
+
<p align="center">
|
73 |
+
<a href="https://damo-nlp-sg.github.io/SeaLLMs/" target="_blank" rel="noopener">Technical Blog</a>
|
74 |
+
|
75 |
+
<a href="https://huggingface.co/SeaLLMs/SeaLLM-7B-v2" target="_blank" rel="noopener"> 🤗 Tech Memo</a>
|
76 |
+
|
77 |
+
<a href="https://huggingface.co/spaces/SeaLLMs/SeaLLM-7B" target="_blank" rel="noopener"> 🤗 DEMO</a>
|
78 |
+
|
79 |
+
<a href="https://github.com/DAMO-NLP-SG/SeaLLMs" target="_blank" rel="noopener">Github</a>
|
80 |
+
|
81 |
+
<a href="https://arxiv.org/pdf/2312.00738.pdf" target="_blank" rel="noopener">Technical Report</a>
|
82 |
+
</p>
|
83 |
+
|
84 |
+
We introduce [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2), the state-of-the-art multilingual LLM for Southeast Asian (SEA) languages 🇬🇧 🇨🇳 🇻🇳 🇮🇩 🇹🇭 🇲🇾 🇰🇭 🇱🇦 🇲🇲 🇵🇭. It is the most significant upgrade since [SeaLLM-13B](https://huggingface.co/SeaLLMs/SeaLLM-13B-Chat), with half the size, outperforming performance across diverse multilingual tasks, from world knowledge, math reasoning, instruction following, etc.
|
85 |
+
|
86 |
+
### Highlights
|
87 |
+
* [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) achieves the **7B-SOTA** on the **Zero-shot CoT GSM8K** task with **78.2** score and outperforms GPT-3.5 in many GSM8K-translated tasks in SEA languages (🇨🇳 🇻🇳 🇮🇩 🇹🇭) as well as MGSM (🇨🇳 🇹🇭). It also surpasses GPT-3.5 in MATH CoT for Thai 🇹🇭.
|
88 |
+
* It scores competitively against GPT-3.5 in many zero-shot CoT commonsense benchmark, with **82.5, 68.3, 80.9** scores on Arc-C, Winogrande, and Hellaswag.
|
89 |
+
* It achieves **7.54** score on the 🇬🇧 **MT-bench**, it ranks 3rd place on the leaderboard for 7B category and is the most outperforming multilingual model.
|
90 |
+
* It scores **45.74** on the VMLU benchmark for Vietnamese 🇻🇳, and is the only open-source multilingual model that can be competitive to monolingual models ([Vistral-7B](https://huggingface.co/Viet-Mistral/Vistral-7B-Chat)) of similar sizes.
|
91 |
+
|
92 |
+
|
93 |
+
### Release and DEMO
|
94 |
+
|
95 |
+
- DEMO: [SeaLLMs/SeaLLM-7B](https://huggingface.co/spaces/SeaLLMs/SeaLLM-7B).
|
96 |
+
- Technical report: [Arxiv: SeaLLMs - Large Language Models for Southeast Asia](https://arxiv.org/pdf/2312.00738.pdf).
|
97 |
+
- Model weights:
|
98 |
+
- [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2).
|
99 |
+
- [SeaLLM-7B-v2-gguf](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf).
|
100 |
+
- [SeaLLM-7B-v2-GGUF (thanks Lonestriker)](https://huggingface.co/LoneStriker/SeaLLM-7B-v2-GGUF). NOTE: use [seallm.preset.json](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf/blob/main/seallm.preset.json) to work properly.
|
101 |
+
- Run locally:
|
102 |
+
- [LM-studio](https://lmstudio.ai/):
|
103 |
+
- [SeaLLM-7B-v2-q4_0](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.q4_0.gguf) and [SeaLLM-7B-v2-q8_0](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.q8_0.gguf).
|
104 |
+
- LM-studio requires this [seallm.preset.json](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf/blob/main/seallm.preset.json) to set chat template properly.
|
105 |
+
- [ollama](https://ollama.ai/) `ollama run nxphi47/seallm-7b-v2:q4_0`
|
106 |
+
- [MLX for Apple Silicon](https://github.com/ml-explore/mlx): [mlx-community/SeaLLM-7B-v2-4bit-mlx](https://huggingface.co/mlx-community/SeaLLM-7B-v2-4bit-mlx)
|
107 |
+
|
108 |
+
<blockquote style="color:red">
|
109 |
+
<p><strong style="color: red">Terms of Use and License</strong>:
|
110 |
+
By using our released weights, codes, and demos, you agree to and comply with the terms and conditions specified in our <a href="https://huggingface.co/SeaLLMs/SeaLLM-Chat-13b/edit/main/LICENSE" target="_blank" rel="noopener">SeaLLMs Terms Of Use</a>.
|
111 |
+
</blockquote>
|
112 |
+
|
113 |
+
> **Disclaimer**:
|
114 |
+
> We must note that even though the weights, codes, and demos are released in an open manner, similar to other pre-trained language models, and despite our best efforts in red teaming and safety fine-tuning and enforcement, our models come with potential risks, including but not limited to inaccurate, misleading or potentially harmful generation.
|
115 |
+
> Developers and stakeholders should perform their own red teaming and provide related security measures before deployment, and they must abide by and comply with local governance and regulations.
|
116 |
+
> In no event shall the authors be held liable for any claim, damages, or other liability arising from the use of the released weights, codes, or demos.
|
117 |
+
|
118 |
+
> The logo was generated by DALL-E 3.
|
119 |
+
|
120 |
+
|
121 |
+
### What's new since SeaLLM-13B-v1 and SeaLLM-7B-v1?
|
122 |
+
|
123 |
+
* SeaLLM-7B-v2 is continue-pretrained from [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-v0.1) and underwent carefully designed tuning with focus in reasoning.
|
124 |
+
|
125 |
+
|
126 |
+
## Evaluation
|
127 |
+
|
128 |
+
|
129 |
+
### Zero-shot CoT Multilingual Math Reasoning
|
130 |
+
|
131 |
+
[SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) achieves with **78.2** score on the GSM8K with zero-shot CoT reasoning, making it the **state of the art** in the realm of 7B models. It also outperforms GPT-3.5 in the same GSM8K benchmark as translated into SEA languages (🇨🇳 🇻🇳 🇮🇩 🇹🇭). [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) also surpasses GPT-3.5 on the Thai-translated MATH benchmark, with **22.4** vs 18.1 scores.
|
132 |
+
|
133 |
+
![fig_sea_math_side_by_side.png](fig_sea_math_side_by_side.png)
|
134 |
+
|
135 |
+
|
136 |
+
<details>
|
137 |
+
<summary>See details on English and translated GSM8K and MATH with zero-shot reasoning</summary>
|
138 |
+
<br>
|
139 |
+
|
140 |
+
| Model | GSM8K<br>en | MATH<br>en | GSM8K<br>zh | MATH<br>zh | GSM8K<br>vi | MATH<br>vi | GSM8K<br>id | MATH<br>id | GSM8K<br>th | MATH<br>th
|
141 |
+
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
142 |
+
| GPT-3.5 | 80.8 | 34.1 | 48.2 | 21.5 | 55 | 26.5 | 64.3 | 26.4 | 35.8 | 18.1
|
143 |
+
| Qwen-14B-chat | 61.4 | 18.4 | 41.6 | 11.8 | 33.6 | 3.6 | 44.7 | 8.6 | 22 | 6
|
144 |
+
| Vistral-7b-chat | 48.2 | 12.5 | | | 48.7 | 3.1 | | | |
|
145 |
+
| Qwen1.5-7B-chat | 56.8 | 15.3 | 40 | 2.7 | 37.7 | 9 | 36.9 | 7.7 | 21.9 |
|
146 |
+
| SeaLLM-7B-v2 | 78.2 | 27.5 | 53.7 | 17.6 | 69.9 | 23.8 | 71.5 | 24.4 | 59.6 | 22.4
|
147 |
+
|
148 |
+
</details>
|
149 |
+
|
150 |
+
Baselines were evaluated using their respective chat-template and system prompts ([Qwen1.5-7B-chat](https://huggingface.co/Qwen/Qwen1.5-7B-Chat/blob/main/tokenizer_config.json), [Vistral](https://huggingface.co/Viet-Mistral/Vistral-7B-Chat)).
|
151 |
+
|
152 |
+
#### Zero-shot MGSM
|
153 |
+
|
154 |
+
[SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) also outperforms GPT-3.5 and Qwen-14B on the multilingual MGSM for Zh and Th.
|
155 |
+
|
156 |
+
| Model | MGSM-Zh | MGSM-Th
|
157 |
+
|-----| ----- | ---
|
158 |
+
| ChatGPT (reported) | 61.2 | 47.2
|
159 |
+
| Qwen-14B-chat | 59.6 | 28
|
160 |
+
| SeaLLM-7B-v2 | **64.8** | **62.4**
|
161 |
+
|
162 |
+
|
163 |
+
### Zero-shot Commonsense Reasoning
|
164 |
+
|
165 |
+
We compare [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) with ChatGPT and Mistral-7B-instruct on various zero-shot commonsense benchmarks (Arc-Challenge, Winogrande and Hellaswag). We use the 2-stage technique in [(Kojima et al., 2023)](https://arxiv.org/pdf/2205.11916.pdf) to grab the answer. Note that we **DID NOT** use "Let's think step-by-step" to invoke explicit CoT.
|
166 |
+
|
167 |
+
| 0-shot reasoning | Arc-Challenge | Winogrande | Hellaswag
|
168 |
+
|-----| ----- | --- | -- |
|
169 |
+
| ChatGPT (reported) | 84.6* | 66.8* | 72.0*
|
170 |
+
| ChatGPT (reproduced)| 84.1 | 63.1 | 79.5
|
171 |
+
| Mistral-7B-Instruct | 68.1 | 56.4 | 45.6
|
172 |
+
| Qwen1.5-7B-chat | 79.3 | 59.4 | 69.3
|
173 |
+
| SeaLLM-7B-v2 | 82.5 | 68.3 | 80.9
|
174 |
+
|
175 |
+
Baselines were evaluated using their respective chat-template and system prompts ([Qwen1.5-7B-chat](https://huggingface.co/Qwen/Qwen1.5-7B-Chat/blob/main/tokenizer_config.json), [Mistral](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)).
|
176 |
+
|
177 |
+
### Multilingual World Knowledge
|
178 |
+
|
179 |
+
|
180 |
+
We evaluate models on 3 benchmarks following the recommended default setups: 5-shot MMLU for En, 3-shot [M3Exam](https://arxiv.org/pdf/2306.05179.pdf) (M3e) for En, Zh, Vi, Id, Th, and zero-shot [VMLU](https://vmlu.ai/) for Vi.
|
181 |
+
|
182 |
+
| Model | Langs | En<br>MMLU | En<br>M3e | Zh<br>M3e | Vi<br>M3e | Vi<br>VMLU | Id<br>M3e | Th<br>M3e
|
183 |
+
|-----| ----- | --- | -- | ----- | ---- | --- | --- | --- |
|
184 |
+
| GPT-3.5 | Multi | 68.90 | 75.46 | 60.20 | 58.64 | 46.32 | 49.27 | 37.41
|
185 |
+
| Vistral-7B-chat | Mono | 56.86 | 67.00 | 44.56 | 54.33 | 50.03 | 36.49 | 25.27
|
186 |
+
| Qwen1.5-7B-chat | Multi | 61.00 | 52.07 | 81.96 | 43.38 | 45.02 | 24.29 | 20.25
|
187 |
+
| SeaLLM-7B-v2 | Multi | 61.89 | 70.91 | 55.43 | 51.15 | 45.74 | 42.25 | 35.52
|
188 |
+
|
189 |
+
|
190 |
+
VMLU reproduce script [here](https://github.com/DAMO-NLP-SG/SeaLLMs/blob/main/evaluation/vmlu/vmlu_run.py). Lm-eval was used to evaluate MMLU.
|
191 |
+
0-shot VMLU scores for baselines were evaluated using their respective chat-template and system prompts ([Qwen1.5-7B-chat](https://huggingface.co/Qwen/Qwen1.5-7B-Chat/blob/main/tokenizer_config.json)).
|
192 |
+
|
193 |
+
|
194 |
+
### MT-Bench
|
195 |
+
|
196 |
+
On the English [MT-bench](https://arxiv.org/abs/2306.05685) metric, SeaLLM-7B-v2 achieves **7.54** score on the MT-bench (3rd place on the leaderboard for 7B category), outperforms many 70B models and is arguably the only one that handles 10 SEA languages.
|
197 |
+
|
198 |
+
Refer to [mt_bench/seallm_7b_v2.jsonl](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2/blob/main/evaluation/mt_bench/seallm_7b_v2.jsonl) for the MT-bench predictions of SeaLLM-7B-v2, and [here](https://github.com/lm-sys/FastChat/issues/3013#issue-2118685341) to reproduce it.
|
199 |
+
|
200 |
+
| Model | Access | Langs | MT-Bench
|
201 |
+
| --- | --- | --- | --- |
|
202 |
+
| GPT-4-turbo | closed | multi | 9.32
|
203 |
+
| GPT-4-0613 | closed | multi | 9.18
|
204 |
+
| Mixtral-8x7b (46B) | open | multi | 8.3
|
205 |
+
| Starling-LM-7B-alpha | open | mono (en) | 8.0
|
206 |
+
| OpenChat-3.5-7B | open | mono (en) | 7.81
|
207 |
+
| **SeaLLM-7B-v2** | **open** | **multi (10+)** | **7.54**
|
208 |
+
| [Qwen-14B](https://huggingface.co/Qwen/Qwen-14B-Chat) | open | multi | 6.96
|
209 |
+
| [Llama-2-70B](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) | open | mono (en) | 6.86
|
210 |
+
| Mistral-7B-instuct | open | mono (en) | 6.84
|
211 |
+
|
212 |
+
|
213 |
+
### Sea-Bench
|
214 |
+
|
215 |
+
Similar to MT-Bench, [Sea-bench](https://huggingface.co/datasets/SeaLLMs/Sea-bench) is a set of categorized instruction test sets to measure models' ability as an assistant that is specifically focused on 9 SEA languages, including non-Latin low-resource languages.
|
216 |
+
|
217 |
+
As shown, the huge improvements come from math-reasoning, reaching GPT-3.5 level of performance.
|
218 |
+
|
219 |
+
![fig_sea_bench_side_by_side.png](fig_sea_bench_side_by_side.png)
|
220 |
+
|
221 |
+
Refer to [sea_bench/seallm_7b_v2.jsonl](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2/blob/main/evaluation/sea_bench/seallm_7b_v2.jsonl) for the Sea-bench predictions of SeaLLM-7B-v2.
|
222 |
+
|
223 |
+
|
224 |
+
### Usage
|
225 |
+
|
226 |
+
#### Instruction format
|
227 |
+
|
228 |
+
```python
|
229 |
+
prompt = """<|im_start|>system
|
230 |
+
You are a helpful assistant.</s><|im_start|>user
|
231 |
+
Hello world</s><|im_start|>assistant
|
232 |
+
Hi there, how can I help?</s>"""
|
233 |
+
|
234 |
+
# NOTE: previous commit has \n between </s> and <|im_start|>, that was incorrect!
|
235 |
+
# <|im_start|> is not a special token.
|
236 |
+
# Transformers chat_template should be consistent with vLLM format below.
|
237 |
+
|
238 |
+
# ! ENSURE 1 and only 1 bos `<s>` at the beginning of sequence
|
239 |
+
print(tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt)))
|
240 |
+
|
241 |
+
'<s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'system', '<0x0A>', 'You', '▁are', '▁a', '▁helpful', '▁assistant', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Hello', '▁world', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>', 'Hi', '▁there', ',', '▁how', '▁can', '▁I', '▁help', '?', '</s>']
|
242 |
+
"""
|
243 |
+
```
|
244 |
+
|
245 |
+
#### Using transformers's chat_template
|
246 |
+
```python
|
247 |
+
|
248 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
249 |
+
|
250 |
+
device = "cuda" # the device to load the model onto
|
251 |
+
|
252 |
+
# use bfloat16 to ensure the best performance.
|
253 |
+
model = AutoModelForCausalLM.from_pretrained("SeaLLMs/SeaLLM-7B-v2", torch_dtype=torch.bfloat16, device_map=device)
|
254 |
+
tokenizer = AutoTokenizer.from_pretrained("SeaLLMs/SeaLLM-7B-v2")
|
255 |
+
|
256 |
+
messages = [
|
257 |
+
{"role": "system", "content": "You are a helpful assistant."},
|
258 |
+
{"role": "user", "content": "Hello world"},
|
259 |
+
{"role": "assistant", "content": "Hi there, how can I help you today?"},
|
260 |
+
{"role": "user", "content": "Explain general relativity in details."}
|
261 |
+
]
|
262 |
+
|
263 |
+
encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
|
264 |
+
print(tokenizer.convert_ids_to_tokens(encodeds[0]))
|
265 |
+
# ['<s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'system', '<0x0A>', 'You', '▁are', '▁a', '▁helpful', '▁assistant', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Hello', '▁world', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>', 'Hi', '▁there', ',', '▁how', '▁can', '▁I', '▁help', '▁you', '▁today', '?', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Ex', 'plain', '▁general', '▁rel', 'ativity', '▁in', '▁details', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>']
|
266 |
+
|
267 |
+
model_inputs = encodeds.to(device)
|
268 |
+
model.to(device)
|
269 |
+
|
270 |
+
generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True, pad_token_id=tokenizer.pad_token_id)
|
271 |
+
decoded = tokenizer.batch_decode(generated_ids)
|
272 |
+
print(decoded[0])
|
273 |
+
|
274 |
+
```
|
275 |
+
|
276 |
+
#### Using vLLM
|
277 |
+
|
278 |
+
```python
|
279 |
+
from vllm import LLM, SamplingParams
|
280 |
+
TURN_TEMPLATE = "<|im_start|>{role}\n{content}</s>"
|
281 |
+
TURN_PREFIX = "<|im_start|>{role}\n"
|
282 |
+
|
283 |
+
# There is no \n between </s> and <|im_start|>.
|
284 |
+
|
285 |
+
def seallm_chat_convo_format(conversations, add_assistant_prefix: bool, system_prompt=None):
|
286 |
+
# conversations: list of dict with key `role` and `content` (openai format)
|
287 |
+
if conversations[0]['role'] != 'system' and system_prompt is not None:
|
288 |
+
conversations = [{"role": "system", "content": system_prompt}] + conversations
|
289 |
+
text = ''
|
290 |
+
for turn_id, turn in enumerate(conversations):
|
291 |
+
prompt = TURN_TEMPLATE.format(role=turn['role'], content=turn['content'])
|
292 |
+
text += prompt
|
293 |
+
if add_assistant_prefix:
|
294 |
+
prompt = TURN_PREFIX.format(role='assistant')
|
295 |
+
text += prompt
|
296 |
+
return text
|
297 |
+
|
298 |
+
sparams = SamplingParams(temperature=0.1, max_tokens=1024, stop=['</s>', '<|im_start|>'])
|
299 |
+
llm = LLM("SeaLLMs/SeaLLM-7B-v2", dtype="bfloat16")
|
300 |
+
|
301 |
+
message = "Explain general relativity in details."
|
302 |
+
prompt = seallm_chat_convo_format(message, True)
|
303 |
+
gen = llm.generate(prompt, sampling_params)
|
304 |
+
|
305 |
+
print(gen[0].outputs[0].text)
|
306 |
+
```
|
307 |
+
|
308 |
+
#### Fine-tuning SeaLLM-7B-v2
|
309 |
+
|
310 |
+
Should follow the chat format and accurately mask out source tokens. Here is an example.
|
311 |
+
|
312 |
+
```python
|
313 |
+
conversations = [
|
314 |
+
{"role": "system", "content": "You are helful assistant."},
|
315 |
+
{"role": "user", "content": "Hello world."},
|
316 |
+
{"role": "assistant", "content": "Hi there, how can I help?"},
|
317 |
+
{"role": "user", "content": "Tell me a joke."},
|
318 |
+
{"role": "assistant", "content": "Why don't scientists trust atoms? Because they make up everything."},
|
319 |
+
]
|
320 |
+
def seallm_7b_v2_tokenize_multi_turns(tokenizer, conversations, add_assistant_prefix=False):
|
321 |
+
"""
|
322 |
+
Inputs:
|
323 |
+
conversations: list of dict following openai format, eg
|
324 |
+
conversations = [
|
325 |
+
{"role": "system", "content": "You are helful assistant."},
|
326 |
+
{"role": "user", "content": "Hello world."},
|
327 |
+
{"role": "assistant", "content": "Hi there, how can I help?"},
|
328 |
+
{"role": "user", "content": "Tell me a joke."},
|
329 |
+
{"role": "assistant", "content": "Why don't scientists trust atoms? Because they make up everything."},
|
330 |
+
]
|
331 |
+
add_assistant_prefix: whether to add assistant_prefix, only for inference decoding
|
332 |
+
Outputs:
|
333 |
+
tokenize_output_sample, {
|
334 |
+
"input_ids": ...
|
335 |
+
"token_type_ids": 1 if train and 0 if masked out (not train)
|
336 |
+
}
|
337 |
+
During training, need to create a labels, with masked-out tokens = -100 to avoid loss computations.
|
338 |
+
labels = sample['input_ids'].clone()
|
339 |
+
labels[sample['token_type_ids'] == 0] = -100
|
340 |
+
"""
|
341 |
+
TURN_TEMPLATE = "<|im_start|>{role}\n{content}</s>"
|
342 |
+
TURN_PREFIX = "<|im_start|>{role}\n"
|
343 |
+
sample = None
|
344 |
+
assistant_prefix_len = None
|
345 |
+
for turn_id, turn in enumerate(conversations):
|
346 |
+
prompt = TURN_TEMPLATE.format(role=turn['role'], content=turn['content'])
|
347 |
+
turn_sample = tokenizer(
|
348 |
+
prompt, padding=False, truncation=False, verbose=False, add_special_tokens=False,
|
349 |
+
return_token_type_ids=True,
|
350 |
+
)
|
351 |
+
if turn['role'] == 'assistant':
|
352 |
+
if assistant_prefix_len is None:
|
353 |
+
assistant_prefix_len = len(tokenizer.encode(TURN_PREFIX.format(role=turn['role']), add_special_tokens=False))
|
354 |
+
turn_sample['token_type_ids'][assistant_prefix_len:] = [1] * (len(turn_sample['input_ids']) - assistant_prefix_len)
|
355 |
+
if sample is None:
|
356 |
+
sample = turn_sample
|
357 |
+
else:
|
358 |
+
for k in turn_sample.keys():
|
359 |
+
sample[k].extend(turn_sample[k])
|
360 |
+
if add_assistant_prefix:
|
361 |
+
assistant_prefix_sample = tokenizer(
|
362 |
+
TURN_PREFIX.format(role="assistant"), padding=False, truncation=False, verbose=False, add_special_tokens=False,
|
363 |
+
return_token_type_ids=True,
|
364 |
+
)
|
365 |
+
for k in sample.keys():
|
366 |
+
sample[k].extend(assistant_prefix_sample[k])
|
367 |
+
if tokenizer.add_bos_token:
|
368 |
+
sample['input_ids'] = [tokenizer.bos_token_id] + sample['input_ids']
|
369 |
+
sample['attention_mask'] = [1] + sample['attention_mask']
|
370 |
+
sample['token_type_ids'] = [sample['token_type_ids'][0]] + sample['token_type_ids']
|
371 |
+
return sample
|
372 |
+
|
373 |
+
# ! testing
|
374 |
+
sample = seallm_7b_v2_tokenize_multi_turns(tokenizer, conversations)
|
375 |
+
print(tokenizer.convert_ids_to_tokens(sample['input_ids']))
|
376 |
+
print(sample['token_type_ids'])
|
377 |
+
# ['<s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'system', '<0x0A>', 'You', '▁are', '▁hel', 'ful', '▁assistant', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Hello', '▁world', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>', 'Hi', '▁there', ',', '▁how', '▁can', '▁I', '▁help', '?', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Tell', '▁me', '▁a', '▁joke', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>', 'Why', '▁don', "'", 't', '▁scientists', '▁trust', '▁atoms', '?', '▁Because', '▁they', '▁make', '▁up', '▁everything', '.', '</s>']
|
378 |
+
# [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
|
379 |
+
|
380 |
+
|
381 |
+
|
382 |
+
```
|
383 |
+
|
384 |
+
|
385 |
+
## Acknowledgement to Our Linguists
|
386 |
+
|
387 |
+
We would like to express our special thanks to our professional and native linguists, Tantong Champaiboon, Nguyen Ngoc Yen Nhi and Tara Devina Putri, who helped build, evaluate, and fact-check our sampled pretraining and SFT dataset as well as evaluating our models across different aspects, especially safety.
|
388 |
+
|
389 |
+
## Citation
|
390 |
+
|
391 |
+
If you find our project useful, we hope you would kindly star our repo and cite our work as follows: Corresponding Author: [[email protected]](mailto:[email protected])
|
392 |
+
|
393 |
+
**Author list and order will change!**
|
394 |
+
|
395 |
+
* `*` and `^` are equal contributions.
|
396 |
+
|
397 |
+
```
|
398 |
+
@article{damonlpsg2023seallm,
|
399 |
+
author = {Xuan-Phi Nguyen*, Wenxuan Zhang*, Xin Li*, Mahani Aljunied*,
|
400 |
+
Zhiqiang Hu, Chenhui Shen^, Yew Ken Chia^, Xingxuan Li, Jianyu Wang,
|
401 |
+
Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang,
|
402 |
+
Chaoqun Liu, Hang Zhang, Lidong Bing},
|
403 |
+
title = {SeaLLMs - Large Language Models for Southeast Asia},
|
404 |
+
year = 2023,
|
405 |
+
Eprint = {arXiv:2312.00738},
|
406 |
+
}
|
407 |
+
```
|
408 |
+
|
409 |
+
|
410 |
+
|