Hugging Face's logo

ZoneTwelve
/
Breeze-7B-Instruct-v1_0-GGUF

Text Generation
GGUF
Chinese
English
Inference Endpoints
conversational
Model card Files Community
ZoneTwelve commited on
Commit
2822e66
•
1 Parent(s): 990919b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +230 -0
README.md CHANGED
@@ -1,3 +1,233 @@
1
  ---
 
2
  license: apache-2.0
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ pipeline_tag: text-generation
3
  license: apache-2.0
4
+ language:
5
+ - zh
6
+ - en
7
  ---
8
+
9
+ # Model Card for MediaTek Research Breeze-7B-Instruct-v1_0
10
+
11
+ MediaTek Research Breeze-7B (hereinafter referred to as Breeze-7B) is a language model family that builds on top of [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-v0.1), specifically intended for Traditional Chinese use.
12
+
13
+ [Breeze-7B-Base](https://huggingface.co/MediaTek-Research/Breeze-7B-Base-v1_0) is the base model for the Breeze-7B series.
14
+ It is suitable for use if you have substantial fine-tuning data to tune it for your specific use case.
15
+
16
+ [Breeze-7B-Instruct](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v1_0) derives from the base model Breeze-7B-Base, making the resulting model amenable to be used as-is for commonly seen tasks.
17
+
18
+ The current release version of Breeze-7B is v1.0, which has undergone a more refined training process compared to Breeze-7B-v0_1, resulting in significantly improved performance in both English and Traditional Chinese.
19
+
20
+ For details of this model please read our [paper](https://arxiv.org/abs/2403.02712).
21
+
22
+ Practicality-wise:
23
+ - Breeze-7B-Base expands the original vocabulary with an additional 30,000 Traditional Chinese tokens. With the expanded vocabulary, and everything else being equal, Breeze-7B operates at twice the inference speed for Traditional Chinese to Mistral-7B and Llama 7B. [See [Inference Performance](#inference-performance).]
24
+ - Breeze-7B-Instruct can be used as is for common tasks such as Q&A, RAG, multi-round chat, and summarization.
25
+
26
+
27
+ Performance-wise:
28
+ - Breeze-7B-Instruct demonstrates impressive performance in benchmarks for Traditional Chinese and English when compared to similar-sized open-source contemporaries such as Taiwan-LLM-7B/13B-chat, QWen(1.5)-7B-Chat, and Yi-6B-Chat. [See [Chat Model Performance](#chat-model-performance).]
29
+
30
+
31
+ *A project by the members (in alphabetical order): Chan-Jan Hsu 許湛然, Chang-Le Liu 劉昶樂, Feng-Ting Liao 廖峰挺, Po-Chun Hsu 許博竣, Yi-Chang Chen 陳宜昌, and the supervisor Da-Shan Shiu 許大山.*
32
+
33
+ ## Demo
34
+
35
+ <a href="https://huggingface.co/spaces/MediaTek-Research/Demo-MR-Breeze-7B" style="color:red;font-weight:bold;">Try Demo Here 👩‍💻🧑🏻‍💻</a>
36
+
37
+
38
+ ## Features
39
+
40
+ - Breeze-7B-Base-v1_0
41
+ - Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
42
+ - 8k-token context length
43
+ - Breeze-7B-Instruct-v1_0
44
+ - Expanding the vocabulary dictionary size from 32k to 62k to better support Traditional Chinese
45
+ - 8k-token context length
46
+ - Multi-turn dialogue (without special handling for harmfulness)
47
+
48
+
49
+ ## Model Details
50
+
51
+ - Breeze-7B-Base-v1_0
52
+ - Finetuned from: [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
53
+ - Model type: Causal decoder-only transformer language model
54
+ - Language: English and Traditional Chinese (zh-tw)
55
+ - Breeze-7B-Instruct-v1_0
56
+ - Finetuned from: [MediaTek-Research/Breeze-7B-Base-v1_0](https://huggingface.co/MediaTek-Research/Breeze-7B-Base-v1_0)
57
+ - Model type: Causal decoder-only transformer language model
58
+ - Language: English and Traditional Chinese (zh-tw)
59
+
60
+ ## Base Model Performance
61
+
62
+ Here we compare Breeze-7B-Base-v1_0 with other open-source base language models of similar parameter size that are widely recognized for their good performance in Chinese.
63
+ **TMMLU+**, **DRCD**, and **Table** source from [MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2).
64
+ [MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2) derives from [TCEval-v1](https://github.com/mtkresearch/MR-Models/tree/main/TC-Eval)
65
+ and [ikala/tmmluplus](https://huggingface.co/datasets/ikala/tmmluplus). **MMLU** sources from [hails/mmlu_no_train](https://huggingface.co/datasets/hails/mmlu_no_train).
66
+ We use the code revised from [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate **TMMLU+**, **DRCD**, **Table**, and **MMLU**. All choice problems adapt the selection by the log-likelihood.
67
+
68
+
69
+ | Models | #Parameters | ↑ TMMLU+ (ACC) | DRCD (EM) | Table (ACC) | MMLU (ACC) |
70
+ |---------------------------------------------- |--------|--------------|-------------|-------------|------------|
71
+ | | |TC, Knowledge |TC, Reasoning|TC, Reasoning|EN, Knowledge|
72
+ | | | 5 shot | 3 shot | 5 shot | 5 shot |
73
+ | [Yi-6B](https://huggingface.co/01-ai/Yi-6B) | 6B | 49.63 | 76.61 | 34.72 | 65.35 |
74
+ | [Qwen1.5-7B](https://huggingface.co/Qwen/Qwen1.5-7B) | 7B | 46.59 | 74.41 | 30.56 | 63.07 |
75
+ | [**Breeze-7B-Base-v1_0**](https://huggingface.co/MediaTek-Research/Breeze-7B-Base-v1_0) | 7B | 42.67 | 80.61 | 31.99 | 61.24 |
76
+ | [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) | 7B | 36.93 | 79.27 | 27.78 | 64.89 |
77
+
78
+ ## Instruction-tuned Model Performance
79
+
80
+ Here we compare Breeze-7B-Instruct-v1_0 with other open-source instruction-tuned language models of similar parameter size that are widely recognized for their good performance in Chinese.
81
+ Also, we listed the benchmark scores of GPT-3.5 Turbo (1106), which represents one of the most widely used high-quality cloud language model API services, for reference.
82
+ **TMMLU+**, **DRCD**, **Table**, and **MT-Bench-tw** source from [MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2).
83
+ [MediaTek-Research/TCEval-v2](https://huggingface.co/datasets/MediaTek-Research/TCEval-v2) derives from [TCEval-v1](https://github.com/mtkresearch/MR-Models/tree/main/TC-Eval)
84
+ and [ikala/tmmluplus](https://huggingface.co/datasets/ikala/tmmluplus). **MMLU** sources from [hails/mmlu_no_train](https://huggingface.co/datasets/hails/mmlu_no_train).
85
+ **MT-Bench** source from [lmsys/mt_bench_human_judgments](https://huggingface.co/datasets/lmsys/mt_bench_human_judgments).
86
+ We use the code revised from [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) to evaluate **TMMLU+**, **DRCD**, **Table**, and **MMLU**. All choice problems adapt the selection by the log-likelihood.
87
+ We use the code revised from [fastchat llm_judge](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge) (GPT4 as judge) to evaluate **MT-Bench-tw** and **MT-Bench**.
88
+
89
+
90
+ | Models | #Parameters | ↑ MT-Bench-tw (Score)| TMMLU+ (ACC) | Table (ACC) | MT-Bench (Score) | MMLU (ACC) |
91
+ |---------------------------------------------------------------------------------------------------------|--------|--------------------|--------------|-------------|------------------|-------------|
92
+ | | |TC, Chat |TC, Knowledge |TC, Reasoning|EN, Chat |EN, Knowledge|
93
+ | | |0 shot | 0 shot | 0 shot |0 shot | 0 shot |
94
+ | [GPT-3.5-Turbo](https://openai.com) | |7.1 | 43.56 | 45.14 |7.9 | 67.09 |
95
+ | [Qwen1.5-7B-Chat](https://huggingface.co/Qwen/Qwen1.5-7B-Chat) | 7B |6.4 | 45.65 | 34.72 |7.6 | 61.85 |
96
+ | [**Breeze-7B-Instruct-v1_0**](https://huggingface.co/MediaTek-Research/Breeze-7B-Instruct-v1_0) | 7B |6.0 | 42.67 | 39.58 |7.4 | 61.73 |
97
+ | [Mistral-7B-v0.2-Instruct](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) | 7B |5.6 | 34.95 | 33.33 |7.6 | 59.97 |
98
+ | [Yi-6B-Chat](https://huggingface.co/01-ai/Yi-6B-Chat) | 6B |5.0 | 44.79 | 25.69 |6.0 | 59.45 |
99
+ | [Taiwan-LLM-13B-v2.0-chat](https://huggingface.co/yentinglin/Taiwan-LLM-13B-v2.0-chat) | 13B |5.0 | 29.47 | 23.61 |N/A* | 50.50 |
100
+ | [Taiwan-LLM-7B-v2.1-chat](https://huggingface.co/yentinglin/Taiwan-LLM-7B-v2.1-chat) | 7B |4.2 | 28.08 | 31.25 |N/A* | 42.72 |
101
+
102
+ \* Taiwan-LLM models respond to multi-turn questions (English) in Traditional Chinese.
103
+
104
+
105
+ | Details on MT-Bench-tw (0 shot):<br/>Models | STEM |Extraction|Reasoning| Math | Coding | Roleplay| Writing |Humanities| AVG |
106
+ |-----------------------------------------------------|---------|---------|---------|---------|---------|---------|---------|----------| --------- |
107
+ | GPT-3.5-Turbo | 7.8 | 6.1 | 5.1 | 6.4 | 6.2 | 8.7 | 7.4 | 9.3 | 7.1 |
108
+ | Qwen1.5-7B-Chat | 9 | 5.6 | 4.7 | 2.8 | 3.7 | 8.0 | 8.0 | 9.4 | 6.4 |
109
+ | **Breeze-7B-Instruct-v1_0** | 7.8 | 5.2 | 4.2 | 4.2 | 4.1 | 7.6 | 5.9 | 9.1 | 6.0 |
110
+ | Mistral-7B-v0.2-Instruct | 6.9 | 4.6 | 4.3 | 3.3 | 4.4 | 7.2 | 6.2 | 7.8 | 5.6 |
111
+ | Yi-6B-Chat | 7.3 | 2.7 | 3.1 | 3.3 | 2.3 | 7.2 | 5.2 | 8.8 | 5.0 |
112
+ | Taiwan-LLM-13B-v2.0-chat | 6.1 | 3.4 | 4.1 | 2.3 | 3.1 | 7.4 | 6.6 | 6.8 | 5.0 |
113
+ | Taiwan-LLM-7B-v2.1-chat | 5.2 | 2.6 | 2.3 | 1.2 | 3.4 | 6.6 | 5.7 | 6.8 | 4.2 |
114
+
115
+
116
+
117
+ | Details on TMMLU+ (0 shot):<br/>Model | STEM | Social Science | Humanities | Other | AVG |
118
+ |-----------------------------------------------------|--------------|----------------|------------|------------|---------|
119
+ | GPT-3.5-Turbo | 41.58 | 48.52 | 40.96 | 43.18 | 43.56 |
120
+ | Qwen1.5-7B-Chat | 41.48 | 51.66 | 44.05 | 45.40 | 45.65 |
121
+ | **Breeze-7B-Instruct-v1_0** | 36.46 | 48.38 | 45.11 | 40.75 | 42.67 |
122
+ | Mistral-7B-v0.2-Instruct | 32.79 | 38.05 | 34.89 | 34.04 | 34.94 |
123
+ | Yi-6B-Chat | 37.80 | 51.74 | 45.36 | 44.25 | 44.79 |
124
+ | Taiwan-LLM-13B-v2.0-chat | 27.74 | 33.69 | 27.03 | 29.43 | 29.47 |
125
+ | Taiwan-LLM-7B-v2.1-chat | 25.58 | 31.76 | 27.36 | 27.61 | 28.08 |
126
+
127
+
128
+
129
+ ## Inference Performance
130
+ In this test, we use the first 700 characters of the [web article](https://health.udn.com/health/story/5976/7699252?from=udn_ch1005_main_index) as the input and ask the model to write the same article again.
131
+ All inferences run on 2 RTX A6000 GPUs (using `vllm`, with a tensor-parallel size of 2).
132
+
133
+ | Models | ↓ Inference Time (sec)|Estimated Max Input Length (Char)|
134
+ |--------------------------------------------------------------------|-------------------|--------------------------|
135
+ | Qwen1.5-7B-Chat | 9.35 | 38.9k |
136
+ | Yi-6B-Chat | 10.62 | 5.2k |
137
+ | **Breeze-7B-Instruct-v1_0** | 10.74 | 11.1k |
138
+ | Mistral-7B-Instruct-v0.2 | 20.48 | 5.1k |
139
+ | Taiwan-LLM-7B-v2.1-chat | 26.26 | 2.2k |
140
+ <!---| Taiwan-LLM-13B-v2.0-chat | 36.80 | 2.2k |--->
141
+
142
+
143
+ <!---## Long-context Performance
144
+ TBD--->
145
+
146
+ ## Use in Transformers
147
+
148
+ First install direct dependencies:
149
+ ```
150
+ pip install transformers torch accelerate
151
+ ```
152
+ If you want faster inference using flash-attention2, you need to install these dependencies:
153
+ ```bash
154
+ pip install packaging ninja
155
+ pip install flash-attn
156
+ ```
157
+ Then load the model in transformers:
158
+ ```python
159
+ from transformers import AutoModelForCausalLM, AutoTokenizer
160
+ import torch
161
+
162
+ # Instruction Model
163
+ model = AutoModelForCausalLM.from_pretrained(
164
+ "MediaTek-Research/Breeze-7B-Instruct-v1_0",
165
+ device_map="auto",
166
+ torch_dtype=torch.bfloat16,
167
+ # attn_implementation="flash_attention_2" # optional
168
+ )
169
+
170
+ # Basemodel
171
+ model = AutoModelForCausalLM.from_pretrained(
172
+ "MediaTek-Research/Breeze-7B-Base-v1_0",
173
+ device_map="auto",
174
+ torch_dtype=torch.bfloat16,
175
+ # attn_implementation="flash_attention_2" # optional
176
+ )
177
+ ```
178
+
179
+ **For Breeze-7B-Instruct**, the structure of the query is
180
+ ```txt
181
+ <s>SYS_PROMPT [INST] QUERY1 [/INST] RESPONSE1 [INST] QUERY2 [/INST]
182
+ ```
183
+ where `SYS_PROMPT`, `QUERY1`, `RESPONSE1`, and `QUERY2` can be provided by the user.
184
+
185
+ The suggested default `SYS_PROMPT` is
186
+ ```txt
187
+ You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan.
188
+ ```
189
+
190
+ We also integrate `chat_template` into [tokenizer_config.json](tokenizer_config.json), so you can `apply_chat_template` to get the prompt.
191
+
192
+ ```python
193
+ >>> from transformers import AutoTokenizer
194
+ >>> tokenizer = AutoTokenizer.from_pretrained("MediaTek-Research/Breeze-7B-Instruct-v1_0")
195
+ >>> chat = [
196
+ ... {"role": "user", "content": "你好，請問你可以完成什麼任務？"},
197
+ ... {"role": "assistant", "content": "你好，我可以幫助您解決各種問題、提供資訊和協助您完成許多不同的任務。例如：回答技術問題、提供建議、翻譯文字、尋找資料或協助您安排行程等。請告訴我如何能幫助您。"},
198
+ ... {"role": "user", "content": "太棒了！"},
199
+ ... ]
200
+ >>> tokenizer.apply_chat_template(chat, tokenize=False)
201
+ "<s>You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan. [INST] 你好，請問你可以完成什麼任務？ [/INST] 你好，我可以幫助您解決各種問題、提供資訊和協助您完成許多不同的任務。例如：回答技術問題、提供建議、翻譯文字、尋找資料或協助您安排行程等。請告訴我如何能幫助您。 [INST] 太棒了！ [/INST] "
202
+ # Tokenized results
203
+ # ['▁', '你好', '，', '請問', '你', '可以', '完成', '什麼', '任務', '？']
204
+ # ['▁', '你好', '，', '我', '可以', '幫助', '您', '解決', '各種', '問題', '、', '提供', '資訊', '和', '協助', '您', '完成', '許多', '不同', '的', '任務', '。', '例如', '：', '回答', '技術', '問題', '、', '提供', '建議', '、', '翻譯', '文字', '、', '尋找', '資料', '或', '協助', '您', '安排', '行程', '等', '。', '請', '告訴', '我', '如何', '能', '幫助', '您', '。']
205
+ # ['▁', '太', '棒', '了', '！']
206
+ ```
207
+
208
+ Text generation can be done by `generate` and `apply_chat_template` functions:
209
+ ```python
210
+ >>> outputs = model.generate(tokenizer.apply_chat_template(chat, return_tensors="pt"),
211
+ >>> # adjust below parameters if necessary
212
+ >>> max_new_tokens=128,
213
+ >>> top_p=0.01,
214
+ >>> top_k=85,
215
+ >>> repetition_penalty=1.1,
216
+ >>> temperature=0.01)
217
+ >>>
218
+ >>> print(tokenizer.decode(outputs[0]))
219
+ ```
220
+
221
+ ## Citation
222
+
223
+ ```
224
+ @article{MediaTek-Research2024breeze7b,
225
+ title={Breeze-7B Technical Report},
226
+ author={Chan-Jan Hsu and Chang-Le Liu and Feng-Ting Liao and Po-Chun Hsu and Yi-Chang Chen and Da-Shan Shiu},
227
+ year={2024},
228
+ eprint={2403.02712},
229
+ archivePrefix={arXiv},
230
+ primaryClass={cs.CL}
231
+ }
232
+ ```
233
+