Anonymous
commited on
Commit
•
d47de13
1
Parent(s):
37a9590
reformat README
Browse files
README.md
CHANGED
|
@@ -5,63 +5,83 @@ license_link: LICENSE
|
|
| 5 |
---
|
| 6 |
<div align="center">
|
| 7 |
|
| 8 |
-
<
|
| 9 |
-
Yi
|
| 10 |
-
</h1>
|
| 11 |
|
| 12 |
</div>
|
| 13 |
|
| 14 |
## Introduction
|
| 15 |
|
| 16 |
-
The **Yi** series models are large language models trained from scratch by
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
## News
|
| 19 |
|
| 20 |
-
- 🎯 **2023/11/02**: The base model of `Yi-6B` and `Yi-34B
|
| 21 |
|
| 22 |
-
## Dependency Installation
|
| 23 |
-
|
| 24 |
-
```shell
|
| 25 |
-
pip install transformers==4.34.0 sentencepiece==0.1.99 accelerate==0.24.1
|
| 26 |
-
```
|
| 27 |
-
|
| 28 |
-
## Generation Demonstration
|
| 29 |
-
|
| 30 |
-
```python
|
| 31 |
-
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 32 |
-
model = AutoModelForCausalLM.from_pretrained("01-ai/Yi-6B", device_map="auto", torch_dtype="auto", trust_remote_code=True)
|
| 33 |
-
tokenizer = AutoTokenizer.from_pretrained("01-ai/Yi-6B", trust_remote_code=True)
|
| 34 |
-
inputs = tokenizer('Please count number for me: 1, 2, 3', return_tensors="pt")
|
| 35 |
-
outputs = model.generate(inputs.input_ids.cuda(), max_new_tokens=256)
|
| 36 |
-
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
| 37 |
-
```
|
| 38 |
|
| 39 |
## Model Performance
|
| 40 |
|
| 41 |
-
| Model | MMLU | CMMLU | C-Eval | GAOKAO | BBH |
|
| 42 |
-
| :------------ | :------: | :------: | :------: | :------: | :------: |
|
| 43 |
-
| | 5-shot | 5-shot | 5-shot | 0-shot | 3-shot@1 | -
|
| 44 |
-
| LLaMA2-34B | 62.6 | - | - | - | 44.1 |
|
| 45 |
-
| LLaMA2-70B | 68.9 | 53.3 | - | 49.8 | 51.2 |
|
| 46 |
-
| Baichuan2-13B | 59.2 | 62.0 | 58.1 | 54.3 | 48.8 |
|
| 47 |
-
| Qwen-14B | 66.3 | 71.0 | 72.1 | 62.5 | 53.4 |
|
| 48 |
-
| Skywork-13B | 62.1 | 61.8 | 60.6 | 68.1 | 41.7 |
|
| 49 |
-
| InternLM-20B | 62.1 | 59.0 | 58.8 | 45.5 | 52.5 |
|
| 50 |
-
| Aquila-34B | 67.8 | 71.4 | 63.1 | - | - | -
|
| 51 |
-
| Falcon-180B | 70.4 | 58.0 | 57.8 | 59.0 | 54.0 |
|
| 52 |
-
| Yi-6B | 63.2 | 75.5 | 72.0 | 72.2 | 42.8 |
|
| 53 |
-
| **Yi-34B** | **76.3** | **83.7** | **81.4** | **82.8** | **54.3** |
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
While benchmarking open-source models, we have observed a disparity between the
|
| 57 |
-
|
| 58 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 59 |
|
| 60 |
## Disclaimer
|
| 61 |
|
| 62 |
-
Although we use data compliance checking algorithms during the training process
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
|
| 64 |
## License
|
| 65 |
|
| 66 |
-
The Yi series
|
| 67 |
-
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
---
|
| 6 |
<div align="center">
|
| 7 |
|
| 8 |
+
<img src="./Yi.svg" width="200px">
|
|
|
|
|
|
|
| 9 |
|
| 10 |
</div>
|
| 11 |
|
| 12 |
## Introduction
|
| 13 |
|
| 14 |
+
The **Yi** series models are large language models trained from scratch by
|
| 15 |
+
developers at [01.AI](https://01.ai/). The first public release contains two
|
| 16 |
+
bilingual(English/Chinese) base models with the parameter sizes of 6B and 34B.
|
| 17 |
+
Both of them are trained with 4K sequence length and can be extended to 32K
|
| 18 |
+
during inference time.
|
| 19 |
|
| 20 |
## News
|
| 21 |
|
| 22 |
+
- 🎯 **2023/11/02**: The base model of `Yi-6B` and `Yi-34B`.
|
| 23 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
|
| 25 |
## Model Performance
|
| 26 |
|
| 27 |
+
| Model | MMLU | CMMLU | C-Eval | GAOKAO | BBH | Common-sense Reasoning | Reading Comprehension | Math & Code |
|
| 28 |
+
| :------------ | :------: | :------: | :------: | :------: | :------: | :--------------------: | :-------------------: | :---------: |
|
| 29 |
+
| | 5-shot | 5-shot | 5-shot | 0-shot | 3-shot@1 | - | - | - |
|
| 30 |
+
| LLaMA2-34B | 62.6 | - | - | - | 44.1 | 69.9 | 68.0 | 26.0 |
|
| 31 |
+
| LLaMA2-70B | 68.9 | 53.3 | - | 49.8 | 51.2 | 71.9 | 69.4 | 36.8 |
|
| 32 |
+
| Baichuan2-13B | 59.2 | 62.0 | 58.1 | 54.3 | 48.8 | 64.3 | 62.4 | 23.0 |
|
| 33 |
+
| Qwen-14B | 66.3 | 71.0 | 72.1 | 62.5 | 53.4 | 73.3 | 72.5 | **39.8** |
|
| 34 |
+
| Skywork-13B | 62.1 | 61.8 | 60.6 | 68.1 | 41.7 | 72.4 | 61.4 | 24.9 |
|
| 35 |
+
| InternLM-20B | 62.1 | 59.0 | 58.8 | 45.5 | 52.5 | 78.3 | - | 30.4 |
|
| 36 |
+
| Aquila-34B | 67.8 | 71.4 | 63.1 | - | - | - | - | - |
|
| 37 |
+
| Falcon-180B | 70.4 | 58.0 | 57.8 | 59.0 | 54.0 | 77.3 | 68.8 | 34.0 |
|
| 38 |
+
| Yi-6B | 63.2 | 75.5 | 72.0 | 72.2 | 42.8 | 72.3 | 68.7 | 19.8 |
|
| 39 |
+
| **Yi-34B** | **76.3** | **83.7** | **81.4** | **82.8** | **54.3** | **80.1** | **76.4** | 37.1 |
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
While benchmarking open-source models, we have observed a disparity between the
|
| 43 |
+
results generated by our pipeline and those reported in public sources (e.g.
|
| 44 |
+
OpenCompass). Upon conducting a more in-depth investigation of this difference,
|
| 45 |
+
we have discovered that various models may employ different prompts,
|
| 46 |
+
post-processing strategies, and sampling techniques, potentially resulting in
|
| 47 |
+
significant variations in the outcomes. Our prompt and post-processing strategy
|
| 48 |
+
remains consistent with the original benchmark, and greedy decoding is employed
|
| 49 |
+
during evaluation without any post-processing for the generated content. For
|
| 50 |
+
scores that were not reported by the original authors (including scores reported
|
| 51 |
+
with different settings), we try to get results with our pipeline.
|
| 52 |
+
|
| 53 |
+
To evaluate the model's capability extensively, we adopted the methodology
|
| 54 |
+
outlined in Llama2. Specifically, we included PIQA, SIQA, HellaSwag, WinoGrande,
|
| 55 |
+
ARC, OBQA, and CSQA to assess common sense reasoning. SquAD, QuAC, and BoolQ
|
| 56 |
+
were incorporated to evaluate reading comprehension. CSQA was exclusively tested
|
| 57 |
+
using a 7-shot setup, while all other tests were conducted with a 0-shot
|
| 58 |
+
configuration. Additionally, we introduced GSM8K (8-shot@1), MATH (4-shot@1),
|
| 59 |
+
HumanEval (0-shot@1), and MBPP (3-shot@1) under the category "Math & Code". Due
|
| 60 |
+
to technical constraints, we did not test Falcon-180 on QuAC and OBQA; the score
|
| 61 |
+
is derived by averaging the scores on the remaining tasks. Since the scores for
|
| 62 |
+
these two tasks are generally lower than the average, we believe that
|
| 63 |
+
Falcon-180B's performance was not underestimated.
|
| 64 |
+
|
| 65 |
+
## Usage
|
| 66 |
+
|
| 67 |
+
Please visit our [github repository](https://github.com/01-ai/) for general
|
| 68 |
+
guidance on how to use this model.
|
| 69 |
|
| 70 |
## Disclaimer
|
| 71 |
|
| 72 |
+
Although we use data compliance checking algorithms during the training process
|
| 73 |
+
to ensure the compliance of the trained model to the best of our ability, due to
|
| 74 |
+
the complexity of the data and the diversity of language model usage scenarios,
|
| 75 |
+
we cannot guarantee that the model will generate correct and reasonable output
|
| 76 |
+
in all scenarios. Please be aware that there is still a risk of the model
|
| 77 |
+
producing problematic outputs. We will not be responsible for any risks and
|
| 78 |
+
issues resulting from misuse, misguidance, illegal usage, and related
|
| 79 |
+
misinformation, as well as any associated data security concerns.
|
| 80 |
|
| 81 |
## License
|
| 82 |
|
| 83 |
+
The **Yi** series models are fully open for academic research and free
|
| 84 |
+
commercial usage. All usage must be adhered to the [Model License
|
| 85 |
+
Agreement](https://huggingface.co/01-ai/Yi-6B/blob/main/LICENSE). To apply for
|
| 86 |
+
the official commercial license, please contact us
|
| 87 |
+
([[email protected]](mailto:[email protected])).
|
Yi.svg
ADDED
|