|
--- |
|
license: mit |
|
--- |
|
<div align="center"> |
|
|
|
<!-- <img src="title.png" alt="LoRA-Flow" width="200"> --> |
|
<!-- ***LORA-Flow*** --> |
|
<b><i style="font-size: 24px;">LORA-Flow</i></b> |
|
|
|
LoRAs and fusion gates for our paper |
|
|
|
<p align="center"> |
|
<a href="https://aclanthology.org/2024.acl-long.695/">Paper</a> • |
|
<a href="https://github.com/pingbowen23/LoRA-Flow"> Github</a> |
|
</p> |
|
</div> |
|
|
|
We released all of our checkpoints used in [LoRA-Flow](https://aclanthology.org/2024.acl-long.695.pdf) which has been accepted to ACL 2024 main conference. |
|
# Summary |
|
> In this repo, we release LoRA modules and the gate of 7B models trained in our paper in HuggingFace format. |
|
# Introduction |
|
LoRA-Flow provides an efficient way to fuse different LoRA modules. The following picture shows our proposed method, we use layer-wise fusion gates to facilitate dynamic LoRA fusion, which project input hidden states of each layer into fusion weights. For more details, please refer to our paper. |
|
 |
|
# Training Data |
|
## Data used for LoRA modules |
|
For the language LoRA modules: we use the 52K training samples from [Okapi](https://aclanthology.org/2023.emnlp-demo.28) for each language, respectively. |
|
|
|
For the math LoRA module: we use [Metamath](https://arxiv.org/abs/2309.12284) that is comprised of 395K mathematical problems and the corresponding solutions in English. |
|
|
|
For the code LoRA module: we use the Magicoder dataset [Magicoder](https://arxiv.org/abs/2312.02120), which consists of 186K code generation problems and the corresponding solutions in English. |
|
|
|
## Data used for gates |
|
We use gates to fuse different LoRA modules. We employ few-shot training and have released our training data. For more details, please refer to our GitHub. |
|
|
|
# Results |
|
We have released the results for LoRAs and LoRA-Flow |
|
|
|
| **Method** | | **MGSM (Math)** | | | | **HumanEval (Code)** | | | | |
|
|-----------------------|-------|-------------------------------|---------|---------|---------|----------------------------------|---------|---------|---------| |
|
| | | **Zh** | **Ru** | **Es** | **Avg.**| **Zh** | **Ru** | **Es** | **Avg.**| |
|
| **Base Model** | | 4.4 | 3.2 | 2.4 | 3.3 | 0.0 | 0.0 | 2.4 | 0.8 | |
|
| **Single LoRA** | Lang | 5.2 | 3.6 | 3.6 | 4.1 | 12.2 | 14.0 | 10.4 | 12.2 | |
|
| | Task | 26.8 | 32.8 | 41.2 | 33.6 | 18.3 | 23.2 | 21.9 | 21.1 | |
|
| **LoRA Fusion** | Avg | 12.8 | 10.4 | 18.4 | 13.9 | 17.1 | 17.7 | 18.3 | 17.7 | |
|
| | LoRA-Hub | 20.8 | 28.4 | 36.8 | 28.7 | 19.5 | 21.3 | 20.1 | 20.3 | |
|
| | LoRA-Flow | **33.2** | **37.6**| **42.0**| **37.6**| **20.7** | **23.8**| **23.2**| **22.6**| |
|
|
|
|
|
# Citation |
|
```bibtex |
|
@inproceedings{wang-etal-2024-lora-flow, |
|
title = "LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks", |
|
author = "Wang, Hanqing and |
|
Ping, Bowen and |
|
Wang, Shuo and |
|
Han, Xu and |
|
Chen, Yun and |
|
Liu, Zhiyuan and |
|
Sun, Maosong", |
|
booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)", |
|
month = aug, |
|
year = "2024", |
|
address = "Bangkok, Thailand", |
|
publisher = "Association for Computational Linguistics", |
|
url = "https://aclanthology.org/2024.acl-long.695", |
|
doi = "10.18653/v1/2024.acl-long.695", |
|
pages = "12871--12882" |
|
} |
|
``` |
|
<!-- [LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks](https://aclanthology.org/2024.acl-long.695) --> |
