|
--- |
|
language: |
|
- en |
|
--- |
|
|
|
# Open-Instruct Flan V2 7B |
|
|
|
This model is a 7B LLaMa model finetuned on the Flan V2 dataset. *Please note this is a model diff - see below for usage instructions*. |
|
|
|
This was trained as part of the paper [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://arxiv.org/abs/2306.04751). |
|
The codebase used to train and evaluate this model can be found at [https://github.com/allenai/open-instruct](https://github.com/allenai/open-instruct). |
|
|
|
This model is licensed under the AI model license given in LICENSE.txt along with the original Llama license (llama_license.txt). |
|
|
|
## Usage |
|
|
|
We assume you have access to a LLaMa model in HF format already. You can find details on getting access and converting the model here: |
|
[https://huggingface.co/docs/transformers/main/model_doc/llama](https://huggingface.co/docs/transformers/main/model_doc/llama) |
|
|
|
Clone [https://github.com/allenai/open-instruct](https://github.com/allenai/open-instruct) and install the required dependencies, or just copy `scripts/weight_diff.py` |
|
and install the minimal requirements listed in `weight-diff-requirements.txt`. Then download or clone this model diff to the same machine. |
|
|
|
Then, run: |
|
```bash |
|
python scripts/weight_diff.py recover --path_raw ${hf_llama_path} --path_tuned ${output_path} --path_diff ${diff_location} |
|
``` |
|
|
|
And you will have a recovered model! Note this takes up a decent amount of RAM, especially for the larger models. |
|
|
|
## Input Format |
|
|
|
The model is trained to use the following format (note the newlines): |
|
``` |
|
<|user|> |
|
Your message here! |
|
<|assistant|> |
|
``` |
|
|
|
For best results, format all inputs in this manner. |
|
|
|
## Performance |
|
|
|
Here is the performance of this model across benchmarks explored in our paper [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://arxiv.org/abs/2306.04751): |
|
|
|
| MMLU 0-shot | MMLU 5-shot | GSM Direct | GSM CoT | BBH Direct | BBH CoT | TydiQA Gold-Passage | TydiQA Closed-book | Codex-Eval Pass@1 | Codex-Eval Pass@10 | AlpacaFarm vs Davinci-003 | Average | |
|
|:-----------:|:-----------:|:----------:|:-------:|:----------:|:-------:|:-------------------:|:------------------:|:-----------------:|:------------------:|:-------------------------:|---------| |
|
| 45.4 | 47.1 | 3.5 | 13.0 | 38.6 | 36.1 | 45.0 | 8.3 | 9.6 | 12.9 | 4.6 | 22.4 | |
|
|
|
If you use this model, please cite our work, the llama paper, and the original dataset: |
|
|
|
``` |
|
@article{camelevaluation, |
|
title={How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources}, |
|
author={Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, Hannaneh Hajishirzi}, |
|
year={2023} |
|
} |
|
``` |
|
|
|
``` |
|
@misc{touvron2023llama, |
|
title={LLaMA: Open and Efficient Foundation Language Models}, |
|
author={Hugo Touvron and Thibaut Lavril and Gautier Izacard and Xavier Martinet and Marie-Anne Lachaux and Timothée Lacroix and Baptiste Rozière and Naman Goyal and Eric Hambro and Faisal Azhar and Aurelien Rodriguez and Armand Joulin and Edouard Grave and Guillaume Lample}, |
|
year={2023}, |
|
eprint={2302.13971}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
|
|
|
``` |
|
@article{longpre2023flan, |
|
title={The Flan Collection: Designing Data and Methods for Effective Instruction Tuning}, |
|
author={Longpre, Shayne and Hou, Le and Vu, Tu and Webson, Albert and Chung, Hyung Won and Tay, Yi and Zhou, Denny and Le, Quoc V and Zoph, Barret and Wei, Jason and others}, |
|
journal={arXiv preprint arXiv:2301.13688}, |
|
year={2023} |
|
} |
|
``` |