---
license: mit
---
<div align="center">

<!-- <img src="title.png" alt="LoRA-Flow" width="200"> -->
<!-- ***LORA-Flow*** -->
<b><i style="font-size: 24px;">LORA-Flow</i></b>

LoRAs and fusion gates for our paper

<p align="center">
 <a href="https://aclanthology.org/2024.acl-long.695/">Paper</a> •
 <a href="https://github.com/pingbowen23/LoRA-Flow"> Github</a>
</p>
</div>

We released all of our checkpoints used in [LoRA-Flow](https://aclanthology.org/2024.acl-long.695.pdf) which has been accepted to ACL 2024 main conference.
# Summary
>  In this repo, we release LoRA modules and the gate of 7B models trained in our paper in HuggingFace format.
# Introduction
LoRA-Flow provides an efficient way to fuse different LoRA modules. The following picture shows our proposed method, we use layer-wise fusion gates to facilitate dynamic LoRA fusion, which project input hidden states of each layer into fusion weights. LoRA-flow can be applied into [Llama-7b backbone](https://huggingface.co/meta-llama/Llama-2-7b) . For more details, please refer to our paper. 
![1.jpg](https://cdn-uploads.huggingface.co/production/uploads/64d99f6cd7e30889c6c477b4/ifiu1FTHilrmUkD4FKkgV.jpeg)
# Training Data
## Data used for LoRA modules
For the language LoRA modules: we use the 52K training samples from [Okapi](https://aclanthology.org/2023.emnlp-demo.28) for each language, respectively.

For the math LoRA module: we use [Metamath](https://arxiv.org/abs/2309.12284) that is comprised of 395K mathematical problems and the corresponding solutions in English.
 
For the code LoRA module: we use the Magicoder dataset [Magicoder](https://arxiv.org/abs/2312.02120), which consists of 186K code generation problems and the corresponding solutions in English.

## Data used for gates
We use gates to fuse different LoRA modules. We employ few-shot training and have released our training data. For more details, please refer to our GitHub.

# Results
We have released the results for LoRAs and LoRA-Flow

| **Method**            |       | **MGSM (Math)**   |         |         |         | **HumanEval (Code)**   |         |         |         |
|-----------------------|-------|-------------------------------|---------|---------|---------|----------------------------------|---------|---------|---------|
|                       |       | **Zh**  | **Ru**  | **Es**  | **Avg.**| **Zh**       | **Ru**  | **Es**  | **Avg.**|
| **Base Model**         |       | 4.4  | 3.2     | 2.4     | 3.3     | 0.0     | 0.0     | 2.4     | 0.8     |
| **Single LoRA**        | Lang  | 5.2 | 3.6     | 3.6     | 4.1     | 12.2     | 14.0    | 10.4    | 12.2    |
|                       | Task  | 26.8     | 32.8    | 41.2    | 33.6    | 18.3     | 23.2    | 21.9    | 21.1    |
| **LoRA Fusion**        | Avg   | 12.8   | 10.4    | 18.4    | 13.9    | 17.1   | 17.7    | 18.3    | 17.7    |
|                       | LoRA-Hub | 20.8   | 28.4    | 36.8    | 28.7    | 19.5   | 21.3    | 20.1    | 20.3    |
|                       | LoRA-Flow | **33.2**    | **37.6**| **42.0**| **37.6**| **20.7**  | **23.8**| **23.2**| **22.6**|


# Citation
```bibtex
@inproceedings{wang-etal-2024-lora-flow,
    title = "LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks",
    author = "Wang, Hanqing  and
      Ping, Bowen  and
      Wang, Shuo  and
      Han, Xu  and
      Chen, Yun  and
      Liu, Zhiyuan  and
      Sun, Maosong",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.695",
    doi = "10.18653/v1/2024.acl-long.695",
    pages = "12871--12882"
}
```
<!-- [LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks](https://aclanthology.org/2024.acl-long.695)  -->