RichardErkhov commited on
Commit
31b3813
1 Parent(s): 09803a0

uploaded readme

Browse files
Files changed (1) hide show
  1. README.md +410 -0
README.md ADDED
@@ -0,0 +1,410 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Quantization made by Richard Erkhov.
2
+
3
+ [Github](https://github.com/RichardErkhov)
4
+
5
+ [Discord](https://discord.gg/pvy7H8DZMG)
6
+
7
+ [Request more models](https://github.com/RichardErkhov/quant_request)
8
+
9
+
10
+ SeaLLM-7B-v2 - GGUF
11
+ - Model creator: https://huggingface.co/SeaLLMs/
12
+ - Original model: https://huggingface.co/SeaLLMs/SeaLLM-7B-v2/
13
+
14
+
15
+ | Name | Quant method | Size |
16
+ | ---- | ---- | ---- |
17
+ | [SeaLLM-7B-v2.Q2_K.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q2_K.gguf) | Q2_K | 2.6GB |
18
+ | [SeaLLM-7B-v2.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.IQ3_XS.gguf) | IQ3_XS | 2.89GB |
19
+ | [SeaLLM-7B-v2.IQ3_S.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.IQ3_S.gguf) | IQ3_S | 3.04GB |
20
+ | [SeaLLM-7B-v2.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q3_K_S.gguf) | Q3_K_S | 3.03GB |
21
+ | [SeaLLM-7B-v2.IQ3_M.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.IQ3_M.gguf) | IQ3_M | 3.14GB |
22
+ | [SeaLLM-7B-v2.Q3_K.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q3_K.gguf) | Q3_K | 3.36GB |
23
+ | [SeaLLM-7B-v2.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q3_K_M.gguf) | Q3_K_M | 3.36GB |
24
+ | [SeaLLM-7B-v2.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q3_K_L.gguf) | Q3_K_L | 3.64GB |
25
+ | [SeaLLM-7B-v2.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.IQ4_XS.gguf) | IQ4_XS | 3.76GB |
26
+ | [SeaLLM-7B-v2.Q4_0.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q4_0.gguf) | Q4_0 | 3.91GB |
27
+ | [SeaLLM-7B-v2.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.IQ4_NL.gguf) | IQ4_NL | 3.96GB |
28
+ | [SeaLLM-7B-v2.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q4_K_S.gguf) | Q4_K_S | 3.94GB |
29
+ | [SeaLLM-7B-v2.Q4_K.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q4_K.gguf) | Q4_K | 4.16GB |
30
+ | [SeaLLM-7B-v2.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q4_K_M.gguf) | Q4_K_M | 4.16GB |
31
+ | [SeaLLM-7B-v2.Q4_1.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q4_1.gguf) | Q4_1 | 4.33GB |
32
+ | [SeaLLM-7B-v2.Q5_0.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q5_0.gguf) | Q5_0 | 4.75GB |
33
+ | [SeaLLM-7B-v2.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q5_K_S.gguf) | Q5_K_S | 4.75GB |
34
+ | [SeaLLM-7B-v2.Q5_K.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q5_K.gguf) | Q5_K | 4.87GB |
35
+ | [SeaLLM-7B-v2.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q5_K_M.gguf) | Q5_K_M | 4.87GB |
36
+ | [SeaLLM-7B-v2.Q5_1.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q5_1.gguf) | Q5_1 | 5.17GB |
37
+ | [SeaLLM-7B-v2.Q6_K.gguf](https://huggingface.co/RichardErkhov/SeaLLMs_-_SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.Q6_K.gguf) | Q6_K | 5.64GB |
38
+
39
+
40
+
41
+
42
+ Original model description:
43
+ ---
44
+ license: other
45
+ license_name: seallms
46
+ license_link: https://huggingface.co/SeaLLMs/SeaLLM-13B-Chat/blob/main/LICENSE
47
+ language:
48
+ - en
49
+ - zh
50
+ - vi
51
+ - id
52
+ - th
53
+ - ms
54
+ - km
55
+ - lo
56
+ - my
57
+ - tl
58
+ tags:
59
+ - multilingual
60
+ - sea
61
+ ---
62
+
63
+ <p align="center">
64
+ <img src="seal_logo.png" width="200" />
65
+ </p>
66
+
67
+ # *SeaLLM-7B-v2* - Large Language Models for Southeast Asia
68
+
69
+ # <strong style="color: red">BIG NEWS: <a href="https://huggingface.co/SeaLLMs/SeaLLM-7B-v2.5">SeaLLM-7B-v2.5</a> is released with state-of-the-art performance in world knowledge and reasoning. SeaLLM-7B-v2 will begin deprecation.</strong>
70
+
71
+
72
+ <p align="center">
73
+ <a href="https://damo-nlp-sg.github.io/SeaLLMs/" target="_blank" rel="noopener">Technical Blog</a>
74
+ &nbsp;&nbsp;
75
+ <a href="https://huggingface.co/SeaLLMs/SeaLLM-7B-v2" target="_blank" rel="noopener"> 🤗 Tech Memo</a>
76
+ &nbsp;&nbsp;
77
+ <a href="https://huggingface.co/spaces/SeaLLMs/SeaLLM-7B" target="_blank" rel="noopener"> 🤗 DEMO</a>
78
+ &nbsp;&nbsp;
79
+ <a href="https://github.com/DAMO-NLP-SG/SeaLLMs" target="_blank" rel="noopener">Github</a>
80
+ &nbsp;&nbsp;
81
+ <a href="https://arxiv.org/pdf/2312.00738.pdf" target="_blank" rel="noopener">Technical Report</a>
82
+ </p>
83
+
84
+ We introduce [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2), the state-of-the-art multilingual LLM for Southeast Asian (SEA) languages 🇬🇧 🇨🇳 🇻🇳 🇮🇩 🇹🇭 🇲🇾 🇰🇭 🇱🇦 🇲🇲 🇵🇭. It is the most significant upgrade since [SeaLLM-13B](https://huggingface.co/SeaLLMs/SeaLLM-13B-Chat), with half the size, outperforming performance across diverse multilingual tasks, from world knowledge, math reasoning, instruction following, etc.
85
+
86
+ ### Highlights
87
+ * [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) achieves the **7B-SOTA** on the **Zero-shot CoT GSM8K** task with **78.2** score and outperforms GPT-3.5 in many GSM8K-translated tasks in SEA languages (🇨🇳 🇻🇳 🇮🇩 🇹🇭) as well as MGSM (🇨🇳 🇹🇭). It also surpasses GPT-3.5 in MATH CoT for Thai 🇹🇭.
88
+ * It scores competitively against GPT-3.5 in many zero-shot CoT commonsense benchmark, with **82.5, 68.3, 80.9** scores on Arc-C, Winogrande, and Hellaswag.
89
+ * It achieves **7.54** score on the 🇬🇧 **MT-bench**, it ranks 3rd place on the leaderboard for 7B category and is the most outperforming multilingual model.
90
+ * It scores **45.74** on the VMLU benchmark for Vietnamese 🇻🇳, and is the only open-source multilingual model that can be competitive to monolingual models ([Vistral-7B](https://huggingface.co/Viet-Mistral/Vistral-7B-Chat)) of similar sizes.
91
+
92
+
93
+ ### Release and DEMO
94
+
95
+ - DEMO: [SeaLLMs/SeaLLM-7B](https://huggingface.co/spaces/SeaLLMs/SeaLLM-7B).
96
+ - Technical report: [Arxiv: SeaLLMs - Large Language Models for Southeast Asia](https://arxiv.org/pdf/2312.00738.pdf).
97
+ - Model weights:
98
+ - [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2).
99
+ - [SeaLLM-7B-v2-gguf](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf).
100
+ - [SeaLLM-7B-v2-GGUF (thanks Lonestriker)](https://huggingface.co/LoneStriker/SeaLLM-7B-v2-GGUF). NOTE: use [seallm.preset.json](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf/blob/main/seallm.preset.json) to work properly.
101
+ - Run locally:
102
+ - [LM-studio](https://lmstudio.ai/):
103
+ - [SeaLLM-7B-v2-q4_0](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.q4_0.gguf) and [SeaLLM-7B-v2-q8_0](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf/blob/main/SeaLLM-7B-v2.q8_0.gguf).
104
+ - LM-studio requires this [seallm.preset.json](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2-gguf/blob/main/seallm.preset.json) to set chat template properly.
105
+ - [ollama](https://ollama.ai/) `ollama run nxphi47/seallm-7b-v2:q4_0`
106
+ - [MLX for Apple Silicon](https://github.com/ml-explore/mlx): [mlx-community/SeaLLM-7B-v2-4bit-mlx](https://huggingface.co/mlx-community/SeaLLM-7B-v2-4bit-mlx)
107
+
108
+ <blockquote style="color:red">
109
+ <p><strong style="color: red">Terms of Use and License</strong>:
110
+ By using our released weights, codes, and demos, you agree to and comply with the terms and conditions specified in our <a href="https://huggingface.co/SeaLLMs/SeaLLM-Chat-13b/edit/main/LICENSE" target="_blank" rel="noopener">SeaLLMs Terms Of Use</a>.
111
+ </blockquote>
112
+
113
+ > **Disclaimer**:
114
+ > We must note that even though the weights, codes, and demos are released in an open manner, similar to other pre-trained language models, and despite our best efforts in red teaming and safety fine-tuning and enforcement, our models come with potential risks, including but not limited to inaccurate, misleading or potentially harmful generation.
115
+ > Developers and stakeholders should perform their own red teaming and provide related security measures before deployment, and they must abide by and comply with local governance and regulations.
116
+ > In no event shall the authors be held liable for any claim, damages, or other liability arising from the use of the released weights, codes, or demos.
117
+
118
+ > The logo was generated by DALL-E 3.
119
+
120
+
121
+ ### What's new since SeaLLM-13B-v1 and SeaLLM-7B-v1?
122
+
123
+ * SeaLLM-7B-v2 is continue-pretrained from [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-v0.1) and underwent carefully designed tuning with focus in reasoning.
124
+
125
+
126
+ ## Evaluation
127
+
128
+
129
+ ### Zero-shot CoT Multilingual Math Reasoning
130
+
131
+ [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) achieves with **78.2** score on the GSM8K with zero-shot CoT reasoning, making it the **state of the art** in the realm of 7B models. It also outperforms GPT-3.5 in the same GSM8K benchmark as translated into SEA languages (🇨🇳 🇻🇳 🇮🇩 🇹🇭). [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) also surpasses GPT-3.5 on the Thai-translated MATH benchmark, with **22.4** vs 18.1 scores.
132
+
133
+ ![fig_sea_math_side_by_side.png](fig_sea_math_side_by_side.png)
134
+
135
+
136
+ <details>
137
+ <summary>See details on English and translated GSM8K and MATH with zero-shot reasoning</summary>
138
+ <br>
139
+
140
+ | Model | GSM8K<br>en | MATH<br>en | GSM8K<br>zh | MATH<br>zh | GSM8K<br>vi | MATH<br>vi | GSM8K<br>id | MATH<br>id | GSM8K<br>th | MATH<br>th
141
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
142
+ | GPT-3.5 | 80.8 | 34.1 | 48.2 | 21.5 | 55 | 26.5 | 64.3 | 26.4 | 35.8 | 18.1
143
+ | Qwen-14B-chat | 61.4 | 18.4 | 41.6 | 11.8 | 33.6 | 3.6 | 44.7 | 8.6 | 22 | 6
144
+ | Vistral-7b-chat | 48.2 | 12.5 | | | 48.7 | 3.1 | | | |
145
+ | Qwen1.5-7B-chat | 56.8 | 15.3 | 40 | 2.7 | 37.7 | 9 | 36.9 | 7.7 | 21.9 |
146
+ | SeaLLM-7B-v2 | 78.2 | 27.5 | 53.7 | 17.6 | 69.9 | 23.8 | 71.5 | 24.4 | 59.6 | 22.4
147
+
148
+ </details>
149
+
150
+ Baselines were evaluated using their respective chat-template and system prompts ([Qwen1.5-7B-chat](https://huggingface.co/Qwen/Qwen1.5-7B-Chat/blob/main/tokenizer_config.json), [Vistral](https://huggingface.co/Viet-Mistral/Vistral-7B-Chat)).
151
+
152
+ #### Zero-shot MGSM
153
+
154
+ [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) also outperforms GPT-3.5 and Qwen-14B on the multilingual MGSM for Zh and Th.
155
+
156
+ | Model | MGSM-Zh | MGSM-Th
157
+ |-----| ----- | ---
158
+ | ChatGPT (reported) | 61.2 | 47.2
159
+ | Qwen-14B-chat | 59.6 | 28
160
+ | SeaLLM-7B-v2 | **64.8** | **62.4**
161
+
162
+
163
+ ### Zero-shot Commonsense Reasoning
164
+
165
+ We compare [SeaLLM-7B-v2](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2) with ChatGPT and Mistral-7B-instruct on various zero-shot commonsense benchmarks (Arc-Challenge, Winogrande and Hellaswag). We use the 2-stage technique in [(Kojima et al., 2023)](https://arxiv.org/pdf/2205.11916.pdf) to grab the answer. Note that we **DID NOT** use "Let's think step-by-step" to invoke explicit CoT.
166
+
167
+ | 0-shot reasoning | Arc-Challenge | Winogrande | Hellaswag
168
+ |-----| ----- | --- | -- |
169
+ | ChatGPT (reported) | 84.6* | 66.8* | 72.0*
170
+ | ChatGPT (reproduced)| 84.1 | 63.1 | 79.5
171
+ | Mistral-7B-Instruct | 68.1 | 56.4 | 45.6
172
+ | Qwen1.5-7B-chat | 79.3 | 59.4 | 69.3
173
+ | SeaLLM-7B-v2 | 82.5 | 68.3 | 80.9
174
+
175
+ Baselines were evaluated using their respective chat-template and system prompts ([Qwen1.5-7B-chat](https://huggingface.co/Qwen/Qwen1.5-7B-Chat/blob/main/tokenizer_config.json), [Mistral](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)).
176
+
177
+ ### Multilingual World Knowledge
178
+
179
+
180
+ We evaluate models on 3 benchmarks following the recommended default setups: 5-shot MMLU for En, 3-shot [M3Exam](https://arxiv.org/pdf/2306.05179.pdf) (M3e) for En, Zh, Vi, Id, Th, and zero-shot [VMLU](https://vmlu.ai/) for Vi.
181
+
182
+ | Model | Langs | En<br>MMLU | En<br>M3e | Zh<br>M3e | Vi<br>M3e | Vi<br>VMLU | Id<br>M3e | Th<br>M3e
183
+ |-----| ----- | --- | -- | ----- | ---- | --- | --- | --- |
184
+ | GPT-3.5 | Multi | 68.90 | 75.46 | 60.20 | 58.64 | 46.32 | 49.27 | 37.41
185
+ | Vistral-7B-chat | Mono | 56.86 | 67.00 | 44.56 | 54.33 | 50.03 | 36.49 | 25.27
186
+ | Qwen1.5-7B-chat | Multi | 61.00 | 52.07 | 81.96 | 43.38 | 45.02 | 24.29 | 20.25
187
+ | SeaLLM-7B-v2 | Multi | 61.89 | 70.91 | 55.43 | 51.15 | 45.74 | 42.25 | 35.52
188
+
189
+
190
+ VMLU reproduce script [here](https://github.com/DAMO-NLP-SG/SeaLLMs/blob/main/evaluation/vmlu/vmlu_run.py). Lm-eval was used to evaluate MMLU.
191
+ 0-shot VMLU scores for baselines were evaluated using their respective chat-template and system prompts ([Qwen1.5-7B-chat](https://huggingface.co/Qwen/Qwen1.5-7B-Chat/blob/main/tokenizer_config.json)).
192
+
193
+
194
+ ### MT-Bench
195
+
196
+ On the English [MT-bench](https://arxiv.org/abs/2306.05685) metric, SeaLLM-7B-v2 achieves **7.54** score on the MT-bench (3rd place on the leaderboard for 7B category), outperforms many 70B models and is arguably the only one that handles 10 SEA languages.
197
+
198
+ Refer to [mt_bench/seallm_7b_v2.jsonl](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2/blob/main/evaluation/mt_bench/seallm_7b_v2.jsonl) for the MT-bench predictions of SeaLLM-7B-v2, and [here](https://github.com/lm-sys/FastChat/issues/3013#issue-2118685341) to reproduce it.
199
+
200
+ | Model | Access | Langs | MT-Bench
201
+ | --- | --- | --- | --- |
202
+ | GPT-4-turbo | closed | multi | 9.32
203
+ | GPT-4-0613 | closed | multi | 9.18
204
+ | Mixtral-8x7b (46B) | open | multi | 8.3
205
+ | Starling-LM-7B-alpha | open | mono (en) | 8.0
206
+ | OpenChat-3.5-7B | open | mono (en) | 7.81
207
+ | **SeaLLM-7B-v2** | **open** | **multi (10+)** | **7.54**
208
+ | [Qwen-14B](https://huggingface.co/Qwen/Qwen-14B-Chat) | open | multi | 6.96
209
+ | [Llama-2-70B](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) | open | mono (en) | 6.86
210
+ | Mistral-7B-instuct | open | mono (en) | 6.84
211
+
212
+
213
+ ### Sea-Bench
214
+
215
+ Similar to MT-Bench, [Sea-bench](https://huggingface.co/datasets/SeaLLMs/Sea-bench) is a set of categorized instruction test sets to measure models' ability as an assistant that is specifically focused on 9 SEA languages, including non-Latin low-resource languages.
216
+
217
+ As shown, the huge improvements come from math-reasoning, reaching GPT-3.5 level of performance.
218
+
219
+ ![fig_sea_bench_side_by_side.png](fig_sea_bench_side_by_side.png)
220
+
221
+ Refer to [sea_bench/seallm_7b_v2.jsonl](https://huggingface.co/SeaLLMs/SeaLLM-7B-v2/blob/main/evaluation/sea_bench/seallm_7b_v2.jsonl) for the Sea-bench predictions of SeaLLM-7B-v2.
222
+
223
+
224
+ ### Usage
225
+
226
+ #### Instruction format
227
+
228
+ ```python
229
+ prompt = """<|im_start|>system
230
+ You are a helpful assistant.</s><|im_start|>user
231
+ Hello world</s><|im_start|>assistant
232
+ Hi there, how can I help?</s>"""
233
+
234
+ # NOTE: previous commit has \n between </s> and <|im_start|>, that was incorrect!
235
+ # <|im_start|> is not a special token.
236
+ # Transformers chat_template should be consistent with vLLM format below.
237
+
238
+ # ! ENSURE 1 and only 1 bos `<s>` at the beginning of sequence
239
+ print(tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt)))
240
+
241
+ '<s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'system', '<0x0A>', 'You', '▁are', '▁a', '▁helpful', '▁assistant', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Hello', '▁world', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>', 'Hi', '▁there', ',', '▁how', '▁can', '▁I', '▁help', '?', '</s>']
242
+ """
243
+ ```
244
+
245
+ #### Using transformers's chat_template
246
+ ```python
247
+
248
+ from transformers import AutoModelForCausalLM, AutoTokenizer
249
+
250
+ device = "cuda" # the device to load the model onto
251
+
252
+ # use bfloat16 to ensure the best performance.
253
+ model = AutoModelForCausalLM.from_pretrained("SeaLLMs/SeaLLM-7B-v2", torch_dtype=torch.bfloat16, device_map=device)
254
+ tokenizer = AutoTokenizer.from_pretrained("SeaLLMs/SeaLLM-7B-v2")
255
+
256
+ messages = [
257
+ {"role": "system", "content": "You are a helpful assistant."},
258
+ {"role": "user", "content": "Hello world"},
259
+ {"role": "assistant", "content": "Hi there, how can I help you today?"},
260
+ {"role": "user", "content": "Explain general relativity in details."}
261
+ ]
262
+
263
+ encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
264
+ print(tokenizer.convert_ids_to_tokens(encodeds[0]))
265
+ # ['<s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'system', '<0x0A>', 'You', '▁are', '▁a', '▁helpful', '▁assistant', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Hello', '▁world', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>', 'Hi', '▁there', ',', '▁how', '▁can', '▁I', '▁help', '▁you', '▁today', '?', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Ex', 'plain', '▁general', '▁rel', 'ativity', '▁in', '▁details', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>']
266
+
267
+ model_inputs = encodeds.to(device)
268
+ model.to(device)
269
+
270
+ generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True, pad_token_id=tokenizer.pad_token_id)
271
+ decoded = tokenizer.batch_decode(generated_ids)
272
+ print(decoded[0])
273
+
274
+ ```
275
+
276
+ #### Using vLLM
277
+
278
+ ```python
279
+ from vllm import LLM, SamplingParams
280
+ TURN_TEMPLATE = "<|im_start|>{role}\n{content}</s>"
281
+ TURN_PREFIX = "<|im_start|>{role}\n"
282
+
283
+ # There is no \n between </s> and <|im_start|>.
284
+
285
+ def seallm_chat_convo_format(conversations, add_assistant_prefix: bool, system_prompt=None):
286
+ # conversations: list of dict with key `role` and `content` (openai format)
287
+ if conversations[0]['role'] != 'system' and system_prompt is not None:
288
+ conversations = [{"role": "system", "content": system_prompt}] + conversations
289
+ text = ''
290
+ for turn_id, turn in enumerate(conversations):
291
+ prompt = TURN_TEMPLATE.format(role=turn['role'], content=turn['content'])
292
+ text += prompt
293
+ if add_assistant_prefix:
294
+ prompt = TURN_PREFIX.format(role='assistant')
295
+ text += prompt
296
+ return text
297
+
298
+ sparams = SamplingParams(temperature=0.1, max_tokens=1024, stop=['</s>', '<|im_start|>'])
299
+ llm = LLM("SeaLLMs/SeaLLM-7B-v2", dtype="bfloat16")
300
+
301
+ message = "Explain general relativity in details."
302
+ prompt = seallm_chat_convo_format(message, True)
303
+ gen = llm.generate(prompt, sampling_params)
304
+
305
+ print(gen[0].outputs[0].text)
306
+ ```
307
+
308
+ #### Fine-tuning SeaLLM-7B-v2
309
+
310
+ Should follow the chat format and accurately mask out source tokens. Here is an example.
311
+
312
+ ```python
313
+ conversations = [
314
+ {"role": "system", "content": "You are helful assistant."},
315
+ {"role": "user", "content": "Hello world."},
316
+ {"role": "assistant", "content": "Hi there, how can I help?"},
317
+ {"role": "user", "content": "Tell me a joke."},
318
+ {"role": "assistant", "content": "Why don't scientists trust atoms? Because they make up everything."},
319
+ ]
320
+ def seallm_7b_v2_tokenize_multi_turns(tokenizer, conversations, add_assistant_prefix=False):
321
+ """
322
+ Inputs:
323
+ conversations: list of dict following openai format, eg
324
+ conversations = [
325
+ {"role": "system", "content": "You are helful assistant."},
326
+ {"role": "user", "content": "Hello world."},
327
+ {"role": "assistant", "content": "Hi there, how can I help?"},
328
+ {"role": "user", "content": "Tell me a joke."},
329
+ {"role": "assistant", "content": "Why don't scientists trust atoms? Because they make up everything."},
330
+ ]
331
+ add_assistant_prefix: whether to add assistant_prefix, only for inference decoding
332
+ Outputs:
333
+ tokenize_output_sample, {
334
+ "input_ids": ...
335
+ "token_type_ids": 1 if train and 0 if masked out (not train)
336
+ }
337
+ During training, need to create a labels, with masked-out tokens = -100 to avoid loss computations.
338
+ labels = sample['input_ids'].clone()
339
+ labels[sample['token_type_ids'] == 0] = -100
340
+ """
341
+ TURN_TEMPLATE = "<|im_start|>{role}\n{content}</s>"
342
+ TURN_PREFIX = "<|im_start|>{role}\n"
343
+ sample = None
344
+ assistant_prefix_len = None
345
+ for turn_id, turn in enumerate(conversations):
346
+ prompt = TURN_TEMPLATE.format(role=turn['role'], content=turn['content'])
347
+ turn_sample = tokenizer(
348
+ prompt, padding=False, truncation=False, verbose=False, add_special_tokens=False,
349
+ return_token_type_ids=True,
350
+ )
351
+ if turn['role'] == 'assistant':
352
+ if assistant_prefix_len is None:
353
+ assistant_prefix_len = len(tokenizer.encode(TURN_PREFIX.format(role=turn['role']), add_special_tokens=False))
354
+ turn_sample['token_type_ids'][assistant_prefix_len:] = [1] * (len(turn_sample['input_ids']) - assistant_prefix_len)
355
+ if sample is None:
356
+ sample = turn_sample
357
+ else:
358
+ for k in turn_sample.keys():
359
+ sample[k].extend(turn_sample[k])
360
+ if add_assistant_prefix:
361
+ assistant_prefix_sample = tokenizer(
362
+ TURN_PREFIX.format(role="assistant"), padding=False, truncation=False, verbose=False, add_special_tokens=False,
363
+ return_token_type_ids=True,
364
+ )
365
+ for k in sample.keys():
366
+ sample[k].extend(assistant_prefix_sample[k])
367
+ if tokenizer.add_bos_token:
368
+ sample['input_ids'] = [tokenizer.bos_token_id] + sample['input_ids']
369
+ sample['attention_mask'] = [1] + sample['attention_mask']
370
+ sample['token_type_ids'] = [sample['token_type_ids'][0]] + sample['token_type_ids']
371
+ return sample
372
+
373
+ # ! testing
374
+ sample = seallm_7b_v2_tokenize_multi_turns(tokenizer, conversations)
375
+ print(tokenizer.convert_ids_to_tokens(sample['input_ids']))
376
+ print(sample['token_type_ids'])
377
+ # ['<s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'system', '<0x0A>', 'You', '▁are', '▁hel', 'ful', '▁assistant', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Hello', '▁world', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>', 'Hi', '▁there', ',', '▁how', '▁can', '▁I', '▁help', '?', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'user', '<0x0A>', 'Tell', '▁me', '▁a', '▁joke', '.', '</s>', '▁<', '|', 'im', '_', 'start', '|', '>', 'ass', 'istant', '<0x0A>', 'Why', '▁don', "'", 't', '▁scientists', '▁trust', '▁atoms', '?', '▁Because', '▁they', '▁make', '▁up', '▁everything', '.', '</s>']
378
+ # [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
379
+
380
+
381
+
382
+ ```
383
+
384
+
385
+ ## Acknowledgement to Our Linguists
386
+
387
+ We would like to express our special thanks to our professional and native linguists, Tantong Champaiboon, Nguyen Ngoc Yen Nhi and Tara Devina Putri, who helped build, evaluate, and fact-check our sampled pretraining and SFT dataset as well as evaluating our models across different aspects, especially safety.
388
+
389
+ ## Citation
390
+
391
+ If you find our project useful, we hope you would kindly star our repo and cite our work as follows: Corresponding Author: [[email protected]](mailto:[email protected])
392
+
393
+ **Author list and order will change!**
394
+
395
+ * `*` and `^` are equal contributions.
396
+
397
+ ```
398
+ @article{damonlpsg2023seallm,
399
+ author = {Xuan-Phi Nguyen*, Wenxuan Zhang*, Xin Li*, Mahani Aljunied*,
400
+ Zhiqiang Hu, Chenhui Shen^, Yew Ken Chia^, Xingxuan Li, Jianyu Wang,
401
+ Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang,
402
+ Chaoqun Liu, Hang Zhang, Lidong Bing},
403
+ title = {SeaLLMs - Large Language Models for Southeast Asia},
404
+ year = 2023,
405
+ Eprint = {arXiv:2312.00738},
406
+ }
407
+ ```
408
+
409
+
410
+