zhaicunqi commited on
Commit
3d0a1c4
1 Parent(s): be3dca9

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +57 -57
README.md CHANGED
@@ -18,7 +18,7 @@ tags:
18
  </h1>
19
  </div>
20
  <div align="center">
21
- 🤗 <a href="https://huggingface.co/qihoo360">Hugging Face</a>&nbsp&nbsp | &nbsp&nbsp
22
  🤖 <a href="https://www.modelscope.cn/profile/qihoo360">ModelScope</a>&nbsp&nbsp | &nbsp&nbsp
23
  💬 <a href="./assets/WeChat.png">WeChat (微信)</a>&nbsp&nbsp
24
  </div>
@@ -29,22 +29,22 @@ tags:
29
 
30
  <br>
31
 
32
- # Models Introduction
33
- 🎉🎉🎉We open-source the 360Zhinao model series:
34
  - **360Zhinao-7B-Base**
35
  - **360Zhinao-7B-Chat-4K**
36
  - **360Zhinao-7B-Chat-32K**
37
  - **360Zhinao-7B-Chat-360K**
38
 
 
39
 
40
- The characteristics of the 360Zhinao open-source models are:
41
- - **Base Model:** Leveraging a high-quality corpus of 3.4 trillion Tokens which mainly consist of Chinese, English and code, we achieved competitive performance on relevant benchmark evaluations of the same model scale.
42
- - **Chat Model:** Powerful chat capabilities and three different sequence lengths of 4K, 32K and 360K. 360K (about 500k Chinese characters) is the longest sequcence length among open-sourced Chinese models until now.
43
 
44
  <br>
45
 
46
  # News and Updates
47
- - 2024.04.12 We release **360Zhinao-7B** 1.0 version, include the base model and three chat model with sequence lengths of 4K, 32K and 360K.
48
 
49
  <br>
50
 
@@ -59,7 +59,7 @@ The characteristics of the 360Zhinao open-source models are:
59
  <br>
60
 
61
  # Download URL
62
- See the following table for this release and download links:
63
  | Size | Model | BF16 | Int4|
64
  |-|-|-|-|
65
  | 7B | 360Zhinao-7B-Base | <a href="https://www.modelscope.cn/models/qihoo360/360Zhinao-7B-Base/summary">🤖</a> <a href="https://huggingface.co/qihoo360/360Zhinao-7B-Base">🤗</a> | |
@@ -71,7 +71,11 @@ See the following table for this release and download links:
71
 
72
  # Model Evaluation
73
  ## Base Model
74
- We evaluate the performance of our model on the OpenCompass evaluation datasets, including C-Eval, AGIEval, MMLU, CMMLU, HellaSwag, MATH, GSM8K, HumanEval, MBPP, BBH, LAMBADA. The ablity evaluated of model include natural language understanding, knowledge, mathematical computation and reasoning, code generation, logical reasoning, etc.
 
 
 
 
75
 
76
  | <div style="width: 100pt">Model</div> | AVG | CEval | AGIEval | MMLU | CMMLU | HellaSwag | MATH | GSM8K | HumanEval | MBPP | BBH | LAMBADA |
77
  |:----------------------|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
@@ -91,28 +95,29 @@ We evaluate the performance of our model on the OpenCompass evaluation datasets,
91
  | Yi-6B | 47.8 | 73 | 44.3 | 64 | **73.5** | 73.1 | 6.3 | 39.9 | 15.2 | 23.6 | 44.9 | 68 |
92
  | **360Zhinao-7B** | 56.15 | **74.11** | 49.49 | **67.44** | 72.38 | **83.05** | 16.38 | 53.83 | 35.98 | 42.4 | 43.95 | **78.59** |
93
 
94
- The above results could be viewed or reproduced on [Opencompass](https://rank.opencompass.org.cn/leaderboard-llm).
95
 
96
  ## Chat Models
97
 
98
- We adopted a two-stage approach to train the long context models.
99
 
100
- **First stage**: We increased RoPE base and extended the context length to 32K.
101
- - Firstly, we performed Continual Pretraining on approximately 5B tokens with a 32K context window.
102
- - Then during the SFT stage, we fine-tuned the model using long data from various sources, including high-quality human-labeled 32K data.
103
 
104
- **Second stage**: We extended the context length to 360K, training with the following data:
105
- - A small amount of high-quality human-labeled super-long data.
106
- - Due to the scarcity of annotated super-long data, we constructed various forms of synthetic data.
107
- - Multi-Doc QA: Similar to [Ziya-Reader](https://arxiv.org/abs/2311.09198), we generated multi-document QA pairs based on 360's database. Multiple QA pairs are constructed for one row of Multi-Doc QA data input, resulting in a multi-turn format and significantly improving the training efficiency.
108
- - Single-Doc QA: Similar to [LLama2 Long](https://arxiv.org/abs/2309.16039), we constructed multi-turn QA data based on different segments within one row of long-text input.
 
 
 
 
109
 
110
  We evaluated our models across various lengths and benchmarks.
111
 
112
  - ### Long Context Benchmarks
113
 
114
 
115
- We evaluated our 32K and 360K models on [LongBench](https://github.com/THUDM/LongBench), a multi-task bilingual benchmark for long contexts. We report results on Chinese tasks that are the most relevant to downstream applications: Single/Multi-Doc QA, Summarization, Few-Shot Learning and Code Completion.
116
 
117
  | Model | Avg | Single-Doc QA | Multi-Doc QA | Summarization | Few-Shot Learning | Code Completion |
118
  | :------------------------ |:---------:|:--------:|:---------:|:---------:|:------------:|:---------:|
@@ -158,18 +163,19 @@ We evaluated our models across various lengths and benchmarks.
158
  <br>
159
 
160
  # Quickstart
161
- Simple examples to illustrate how to use 360Zhinao-7B-Base and 360Zhinao-7B-Chat quickly using 🤖 ModelScope and 🤗 Transformers
162
 
163
  ## Dependency Installation
164
- - python 3.8 and above
165
- - pytorch 2.0 and above
166
- - transformers 4.37.2 and above
167
- - CUDA 11.4 and above are recommended.
168
 
169
  ```shell
170
  pip install -r requirements.txt
171
  ```
172
- We recommend installing Flash-Attention (which currently supports flash attention 2) to increase your performance and reduce your memory footprint. (flash-attention is optional and will work without installation)
 
173
 
174
  >flash-attn >= 2.3.6
175
  ```shell
@@ -179,7 +185,6 @@ FLASH_ATTENTION_FORCE_BUILD=TRUE pip install flash-attn==2.3.6
179
  ## 🤗 Transformers
180
  ### Demonstration of Base Model Inference
181
 
182
- This code demonstrates fast inference with 360Zhinao-7B-Base models using transformers.
183
  ```python
184
  from transformers import AutoTokenizer, AutoModelForCausalLM
185
  from transformers.generation import GenerationConfig
@@ -207,7 +212,6 @@ print("outputs:\n", tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))
207
  ```
208
  ### Demonstration of Chat Model Inference
209
 
210
- This code demo uses transformers to quickly use the 360Zhinao-7B-Chat-4K model for inference.
211
  ```python
212
  from transformers import AutoTokenizer, AutoModelForCausalLM
213
  from transformers.generation import GenerationConfig
@@ -244,8 +248,6 @@ print(messages)
244
  ## 🤖 ModelScope
245
  ### Demonstration of Base Model Inference
246
 
247
- This code demonstrates using ModelScope to quickly use the 360Zhinao-7B-Base model for inference.
248
-
249
  ```python
250
  from modelscope import AutoModelForCausalLM, AutoTokenizer
251
  from modelscope import GenerationConfig
@@ -274,8 +276,6 @@ print("outputs:\n", tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))
274
 
275
  ### Demonstration of Chat Model Inference
276
 
277
- This code demonstrates using ModelScope to quickly use the 360Zhinao-7B-Chat-4K model for inference.
278
-
279
  ```python
280
  from modelscope import AutoModelForCausalLM, AutoTokenizer
281
  from modelscope import GenerationConfig
@@ -310,7 +310,8 @@ print(messages)
310
  ```
311
 
312
  ## CLI Demo
313
- Use terminal interaction for a fast experience
 
314
  ```shell
315
  python cli_demo.py
316
  ```
@@ -319,7 +320,7 @@ python cli_demo.py
319
  <p>
320
 
321
  ## Web Demo
322
- You can also use web interaction for a quick experience
323
  ```shell
324
  streamlit run web_demo.py
325
  ```
@@ -328,12 +329,12 @@ streamlit run web_demo.py
328
  <p>
329
 
330
  ## API Demo
331
- Start command
332
  ```shell
333
  python openai_api.py
334
  ```
335
 
336
- Request parameter
337
  ```shell
338
  curl 'http://localhost:8360/v1/chat/completions' \
339
  -H 'Content-Type: application/json' \
@@ -355,23 +356,23 @@ curl 'http://localhost:8360/v1/chat/completions' \
355
 
356
  # Model Inference
357
  ## Quantization
358
- We provide quantization schemes based on AutoGPTQ and open source the Int4 quantization models.
359
 
360
  ## Deployment
361
  ### vLLM Installation
362
- If you want to deploy and accelerate inference, we recommend using `vLLM==0.3.3`。
363
 
364
- If you are using **CUDA 12.1 and PyTorch 2.1**, you can install vLLM directly with the following command.
365
  ```shell
366
  pip install vllm==0.3.3
367
  ```
368
 
369
- Otherwise, please refer to the official vLLM [Installation Instructions](https://docs.vllm.ai/en/latest/getting_started/installation.html)
370
 
371
- >Once the installation is complete, you will need to do the following
372
- 1. Copy the vllm/zhinao.py file to the vllm/model_executor/models directory corresponding to your env environment.
373
- 2. Copy the vllm/serving_chat.py file to the vllm/entrypoints/openai corresponding to your env environment.
374
- 3. Then add a line to vllm/model_executor/models/\_\_init\_\_.py
375
 
376
  ```shell
377
  "ZhinaoForCausalLM": ("zhinao", "ZhinaoForCausalLM"),
@@ -379,7 +380,7 @@ Otherwise, please refer to the official vLLM [Installation Instructions](https:/
379
 
380
  ### vLLM Service Start
381
 
382
- Starting the service
383
  ```shell
384
  python -m vllm.entrypoints.openai.api_server \
385
  --served-model-name 360Zhinao-7B-Chat-4K \
@@ -391,7 +392,7 @@ python -m vllm.entrypoints.openai.api_server \
391
  --port 8360
392
  ```
393
 
394
- Use curl to request the service
395
  ```shell
396
  curl http://localhost:8360/v1/chat/completions \
397
  -H "Content-Type: application/json" \
@@ -414,7 +415,7 @@ curl http://localhost:8360/v1/chat/completions \
414
  ]
415
  }'
416
  ```
417
- Use python to request the service
418
  ```python
419
  from openai import OpenAI
420
  openai_api_key = "EMPTY"
@@ -442,16 +443,15 @@ chat_response = client.chat.completions.create(
442
  print("Chat response:", chat_response)
443
  ```
444
 
445
- > Notice: If you need to enable repetition penalty, recommended to use *presence_penalty* and *frequency_penalty* parameters.
446
 
447
- >
448
 
449
  <br>
450
 
451
  # Model Finetune
452
  ## Training data
453
 
454
- Training Data: data/training_data_sample.json. The sample data is 10,000 pieces sampled from [multiturn_chat_0.8M](https://huggingface.co/datasets/BelleGroup/multiturn_chat_0.8M) and format converted.
455
 
456
  Data Format:
457
  ```json
@@ -475,7 +475,7 @@ Data Format:
475
  }
476
  ]
477
  ```
478
- ## Fine-tuning scripts
479
  ```shell
480
  set -x
481
 
@@ -531,15 +531,15 @@ deepspeed --hostfile ${HOSTFILE} \
531
  ```shell
532
  bash finetune/ds_finetune.sh
533
  ```
534
- - By configuring the **hostfile**, single-machine and multi-machine training can be realized.
535
- - By configuring **ds_config**, realize zero2 and zero3 training
536
- - By configuring the **fp16**、**bf16** realize mixed precision training, bf16 is recommended to be consistent with the pre-trained model.
537
- - By configuring **is_concat**, Whether the training data is concatenated or not is controlled. When the magnitude of the training data is large, the training efficiency can be improved by concatenation.
538
 
539
  <br>
540
 
541
  # License
542
 
543
- The source code of this warehouse follows the open source license Apache 2.0.
544
 
545
- The 360 ​Zhinao open source model supports commercial use. If you need to use this model and its derivative models for commercial purposes, please contact us via email ([email protected]) to apply. For the specific license agreement, please see [360 Zhinao Open Source Model License](https://github.com/Qihoo360/360zhinao/blob/main/360%E6%99%BA%E8%84%91%E5%BC%80%E6%BA%90%E6%A8%A1%E5%9E%8B%E8%AE%B8%E5%8F%AF%E8%AF%81.txt).
 
18
  </h1>
19
  </div>
20
  <div align="center">
21
+ 🤗 <a href="https://huggingface.co/qihoo360">HuggingFace</a>&nbsp&nbsp | &nbsp&nbsp
22
  🤖 <a href="https://www.modelscope.cn/profile/qihoo360">ModelScope</a>&nbsp&nbsp | &nbsp&nbsp
23
  💬 <a href="./assets/WeChat.png">WeChat (微信)</a>&nbsp&nbsp
24
  </div>
 
29
 
30
  <br>
31
 
32
+ # Introduction
33
+ 🎉🎉🎉 We released the 360Zhinao model series:
34
  - **360Zhinao-7B-Base**
35
  - **360Zhinao-7B-Chat-4K**
36
  - **360Zhinao-7B-Chat-32K**
37
  - **360Zhinao-7B-Chat-360K**
38
 
39
+ Notable features of our 360Zhinao models are:
40
 
41
+ - **Base Model:** Leveraging a high-quality corpus of 3.4 trillion tokens consisting of mainly Chinese, English and code, we achieved competitive performance on relevant benchmarks against other 7B models.
42
+ - **Chat Models:** Powerful chat capabilities and three context lengths of 4K, 32K and 360K. 360K (around 500k Chinese characters) is the longest context length among Chinese open-sourced models upon release (Apr. 11, 2024).
 
43
 
44
  <br>
45
 
46
  # News and Updates
47
+ - [2024.04.12] We released **360Zhinao-7B** v1.0, including the base model and three chat models with context lengths 4K, 32K and 360K.
48
 
49
  <br>
50
 
 
59
  <br>
60
 
61
  # Download URL
62
+
63
  | Size | Model | BF16 | Int4|
64
  |-|-|-|-|
65
  | 7B | 360Zhinao-7B-Base | <a href="https://www.modelscope.cn/models/qihoo360/360Zhinao-7B-Base/summary">🤖</a> <a href="https://huggingface.co/qihoo360/360Zhinao-7B-Base">🤗</a> | |
 
71
 
72
  # Model Evaluation
73
  ## Base Model
74
+ We evaluate our model on [OpenCompass](https://opencompass.org.cn/home), more specifically on C-Eval, AGIEval, MMLU, CMMLU, HellaSwag, MATH, GSM8K, HumanEval, MBPP, BBH and LAMBADA.
75
+ These benchmarks test the model on
76
+ natural language understanding, knowledge, mathematics, code generation and logical reasoning, etc.
77
+
78
+ Results are listed as follows and could be viewed or reproduced on [OpenCompass leaderboard](https://rank.opencompass.org.cn/leaderboard-llm).
79
 
80
  | <div style="width: 100pt">Model</div> | AVG | CEval | AGIEval | MMLU | CMMLU | HellaSwag | MATH | GSM8K | HumanEval | MBPP | BBH | LAMBADA |
81
  |:----------------------|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
 
95
  | Yi-6B | 47.8 | 73 | 44.3 | 64 | **73.5** | 73.1 | 6.3 | 39.9 | 15.2 | 23.6 | 44.9 | 68 |
96
  | **360Zhinao-7B** | 56.15 | **74.11** | 49.49 | **67.44** | 72.38 | **83.05** | 16.38 | 53.83 | 35.98 | 42.4 | 43.95 | **78.59** |
97
 
 
98
 
99
  ## Chat Models
100
 
101
+ The 4K and 32K models are trained separately with the same 4K SFT data.
102
 
103
+ To train the long-context models, we adopted a two-stage approach.
 
 
104
 
105
+ **First stage**: We increased RoPE base and extended the context length to 32K.
106
+ - Firstly, we performed Continual Pretraining on approximately 5B tokens with a 32K context window.
107
+ - Then during the SFT stage, we finetuned the model using long data from various sources, including high-quality human-labeled 32K data.
108
+
109
+ **Second stage**: We extended the context length to 360K, training with the following data:
110
+ - A small amount of high-quality human-labeled super-long data.
111
+ - Due to the scarcity of annotated super-long data, we constructed various forms of synthetic data.
112
+ - Multi-Doc QA: Similar to [Ziya-Reader](https://arxiv.org/abs/2311.09198), we generated multi-document QA pairs based on 360's database. Multiple QA pairs are constructed for one row of Multi-Doc QA data input, resulting in a multi-turn format and significantly improving the training efficiency.
113
+ - Single-Doc QA: Similar to [LLama2 Long](https://arxiv.org/abs/2309.16039), we constructed multi-turn QA data based on different segments within one row of long-text input.
114
 
115
  We evaluated our models across various lengths and benchmarks.
116
 
117
  - ### Long Context Benchmarks
118
 
119
 
120
+ We evaluated our 32K and 360K models on [LongBench](https://github.com/THUDM/LongBench), a multi-task bilingual benchmark for long contexts. We report results on **Chinese** tasks most relevant to downstream applications: Single/Multi-Doc QA, Summarization, Few-Shot Learning and Code Completion.
121
 
122
  | Model | Avg | Single-Doc QA | Multi-Doc QA | Summarization | Few-Shot Learning | Code Completion |
123
  | :------------------------ |:---------:|:--------:|:---------:|:---------:|:------------:|:---------:|
 
163
  <br>
164
 
165
  # Quickstart
166
+ We provide simple examples illustrating the use of 360Zhinao-7B-Base and 360Zhinao-7B-Chat on 🤖ModelScope and 🤗Transformers.
167
 
168
  ## Dependency Installation
169
+ - python >= 3.8
170
+ - pytorch >= 2.0
171
+ - transformers >= 4.37.2
172
+ - CUDA >= 11.4
173
 
174
  ```shell
175
  pip install -r requirements.txt
176
  ```
177
+
178
+ Optionally, we recommend installing Flash-Attention 2 to improve performance and reduce memory footprint.
179
 
180
  >flash-attn >= 2.3.6
181
  ```shell
 
185
  ## 🤗 Transformers
186
  ### Demonstration of Base Model Inference
187
 
 
188
  ```python
189
  from transformers import AutoTokenizer, AutoModelForCausalLM
190
  from transformers.generation import GenerationConfig
 
212
  ```
213
  ### Demonstration of Chat Model Inference
214
 
 
215
  ```python
216
  from transformers import AutoTokenizer, AutoModelForCausalLM
217
  from transformers.generation import GenerationConfig
 
248
  ## 🤖 ModelScope
249
  ### Demonstration of Base Model Inference
250
 
 
 
251
  ```python
252
  from modelscope import AutoModelForCausalLM, AutoTokenizer
253
  from modelscope import GenerationConfig
 
276
 
277
  ### Demonstration of Chat Model Inference
278
 
 
 
279
  ```python
280
  from modelscope import AutoModelForCausalLM, AutoTokenizer
281
  from modelscope import GenerationConfig
 
310
  ```
311
 
312
  ## CLI Demo
313
+ Use terminal for command-line interface:
314
+
315
  ```shell
316
  python cli_demo.py
317
  ```
 
320
  <p>
321
 
322
  ## Web Demo
323
+
324
  ```shell
325
  streamlit run web_demo.py
326
  ```
 
329
  <p>
330
 
331
  ## API Demo
332
+ Launch api:
333
  ```shell
334
  python openai_api.py
335
  ```
336
 
337
+ Then request with parameters:
338
  ```shell
339
  curl 'http://localhost:8360/v1/chat/completions' \
340
  -H 'Content-Type: application/json' \
 
356
 
357
  # Model Inference
358
  ## Quantization
359
+ We provide quantization schemes based on AutoGPTQ and release the Int4 quantization models.
360
 
361
  ## Deployment
362
  ### vLLM Installation
363
+ We recommend using `vLLM==0.3.3`.
364
 
365
+ If you are using **CUDA 12.1 and PyTorch 2.1**, you can install vLLM directly with:
366
  ```shell
367
  pip install vllm==0.3.3
368
  ```
369
 
370
+ Otherwise, please refer to the official vLLM [Installation Instructions](https://docs.vllm.ai/en/latest/getting_started/installation.html).
371
 
372
+ After installation, perform the following steps:
373
+ 1. Copy `vllm/zhinao.py` into `vllm/model_executor/models` in your vllm installation directory (in python/conda env).
374
+ 2. Copy `vllm/serving_chat.py` into `vllm/entrypoints/openai` in your vllm installation directory.
375
+ 3. Then add a line in `vllm/model_executor/models/__init__.py`
376
 
377
  ```shell
378
  "ZhinaoForCausalLM": ("zhinao", "ZhinaoForCausalLM"),
 
380
 
381
  ### vLLM Service Start
382
 
383
+ Start the service:
384
  ```shell
385
  python -m vllm.entrypoints.openai.api_server \
386
  --served-model-name 360Zhinao-7B-Chat-4K \
 
392
  --port 8360
393
  ```
394
 
395
+ Use curl to request the service:
396
  ```shell
397
  curl http://localhost:8360/v1/chat/completions \
398
  -H "Content-Type: application/json" \
 
415
  ]
416
  }'
417
  ```
418
+ Use python to request the service:
419
  ```python
420
  from openai import OpenAI
421
  openai_api_key = "EMPTY"
 
443
  print("Chat response:", chat_response)
444
  ```
445
 
446
+ > If you need to enable repetition penalty, we recommend setting `presence_penalty` and `frequency_penalty` instead of `repetition_penalty`.
447
 
 
448
 
449
  <br>
450
 
451
  # Model Finetune
452
  ## Training data
453
 
454
+ Training Data: `data/training_data_sample.json`. This example data has 10,000 rows sampled from [multiturn_chat_0.8M](https://huggingface.co/datasets/BelleGroup/multiturn_chat_0.8M) with converted format.
455
 
456
  Data Format:
457
  ```json
 
475
  }
476
  ]
477
  ```
478
+ ## Finetuning scripts
479
  ```shell
480
  set -x
481
 
 
531
  ```shell
532
  bash finetune/ds_finetune.sh
533
  ```
534
+ - Configuring `HOSTFILE` switches between single-machine and multi-machine training.
535
+ - configuring `ds_config` switches between zero1, zero2 and zero3.
536
+ - `fp16, bf16` could configure mixed precision training. bf16 is recommended to be consistent with the pretrained model.
537
+ - `is_concat` configures whether the training data is concatenated or not.
538
 
539
  <br>
540
 
541
  # License
542
 
543
+ The source code of this repository follows the open-source license Apache 2.0.
544
 
545
+ 360​Zhinao open-source models support commercial use. If you wish to use these models or continue training them for commercial purposes, please contact us via email ([email protected]) to apply. For the specific license agreement, please see [<<360 Zhinao Open-Source Model License>>](https://github.com/Qihoo360/360zhinao/blob/main/360%E6%99%BA%E8%84%91%E5%BC%80%E6%BA%90%E6%A8%A1%E5%9E%8B%E8%AE%B8%E5%8F%AF%E8%AF%81.txt).