File size: 3,953 Bytes
762668d 4385b31 762668d b026360 4385b31 3c23448 762668d 4827ef5 b026360 4827ef5 b026360 4827ef5 b026360 4827ef5 b026360 4827ef5 b058e42 762668d 4385b31 4827ef5 4385b31 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
---
license: mit
---
<div align="center">
<!-- <img src="title.png" alt="LoRA-Flow" width="200"> -->
<!-- ***LORA-Flow*** -->
<b><i style="font-size: 24px;">LORA-Flow</i></b>
LoRAs and fusion gates for our paper
<p align="center">
<a href="https://aclanthology.org/2024.acl-long.695/">Paper</a> •
<a href="https://github.com/pingbowen23/LoRA-Flow"> Github</a>
</p>
</div>
We released all of our checkpoints used in [LoRA-Flow](https://aclanthology.org/2024.acl-long.695.pdf) which has been accepted to ACL 2024 main conference.
# Summary
> In this repo, we release LoRA modules and the gate of 7B models trained in our paper in HuggingFace format.
# Introduction
LoRA-Flow provides an efficient way to fuse different LoRA modules. The following picture shows our proposed method, we use layer-wise fusion gates to facilitate dynamic LoRA fusion, which project input hidden states of each layer into fusion weights. LoRA-flow can be applied into [Llama-7b backbone](https://huggingface.co/meta-llama/Llama-2-7b) . For more details, please refer to our paper.
# Training Data
## Data used for LoRA modules
For the language LoRA modules: we use the 52K training samples from [Okapi](https://aclanthology.org/2023.emnlp-demo.28) for each language, respectively.
For the math LoRA module: we use [Metamath](https://arxiv.org/abs/2309.12284) that is comprised of 395K mathematical problems and the corresponding solutions in English.
For the code LoRA module: we use the Magicoder dataset [Magicoder](https://arxiv.org/abs/2312.02120), which consists of 186K code generation problems and the corresponding solutions in English.
## Data used for gates
We use gates to fuse different LoRA modules. We employ few-shot training and have released our training data. For more details, please refer to our GitHub.
# Results
We have released the results for LoRAs and LoRA-Flow
| **Method** | | **MGSM (Math)** | | | | **HumanEval (Code)** | | | |
|-----------------------|-------|-------------------------------|---------|---------|---------|----------------------------------|---------|---------|---------|
| | | **Zh** | **Ru** | **Es** | **Avg.**| **Zh** | **Ru** | **Es** | **Avg.**|
| **Base Model** | | 4.4 | 3.2 | 2.4 | 3.3 | 0.0 | 0.0 | 2.4 | 0.8 |
| **Single LoRA** | Lang | 5.2 | 3.6 | 3.6 | 4.1 | 12.2 | 14.0 | 10.4 | 12.2 |
| | Task | 26.8 | 32.8 | 41.2 | 33.6 | 18.3 | 23.2 | 21.9 | 21.1 |
| **LoRA Fusion** | Avg | 12.8 | 10.4 | 18.4 | 13.9 | 17.1 | 17.7 | 18.3 | 17.7 |
| | LoRA-Hub | 20.8 | 28.4 | 36.8 | 28.7 | 19.5 | 21.3 | 20.1 | 20.3 |
| | LoRA-Flow | **33.2** | **37.6**| **42.0**| **37.6**| **20.7** | **23.8**| **23.2**| **22.6**|
# Citation
```bibtex
@inproceedings{wang-etal-2024-lora-flow,
title = "LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks",
author = "Wang, Hanqing and
Ping, Bowen and
Wang, Shuo and
Han, Xu and
Chen, Yun and
Liu, Zhiyuan and
Sun, Maosong",
booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = aug,
year = "2024",
address = "Bangkok, Thailand",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.acl-long.695",
doi = "10.18653/v1/2024.acl-long.695",
pages = "12871--12882"
}
```
<!-- [LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks](https://aclanthology.org/2024.acl-long.695) --> |