README.md

---
license: mit
---
<div align="center">

<!-- <img src="title.png" alt="LoRA-Flow" width="200"> -->
<!-- ***LORA-Flow*** -->
<b><i style="font-size: 24px;">LORA-Flow</i></b>

LoRAs and fusion gates for our paper

<p align="center">
 <a href="https://aclanthology.org/2024.acl-long.695/">Paper</a> •
 <a href="https://github.com/pingbowen23/LoRA-Flow"> Github</a>
</p>
</div>

We released all of our checkpoints used in [LoRA-Flow](https://aclanthology.org/2024.acl-long.695.pdf) which has been accepted to ACL 2024 main conference.
# Summary
>  In this repo, we release LoRA modules and the gate of 7B models trained in our paper in HuggingFace format.
# Introduction
LoRA-Flow provides an efficient way to fuse different LoRA modules. The following picture shows our proposed method, we use layer-wise fusion gates to facilitate dynamic LoRA fusion, which project input hidden states of each layer into fusion weights. LoRA-flow can be applied into [Llama-7b backbone](https://huggingface.co/meta-llama/Llama-2-7b) . For more details, please refer to our paper. 
![1.jpg](https://cdn-uploads.huggingface.co/production/uploads/64d99f6cd7e30889c6c477b4/ifiu1FTHilrmUkD4FKkgV.jpeg)
# Training Data
## Data used for LoRA modules
For the language LoRA modules: we use the 52K training samples from [Okapi](https://aclanthology.org/2023.emnlp-demo.28) for each language, respectively.

For the math LoRA module: we use [Metamath](https://arxiv.org/abs/2309.12284) that is comprised of 395K mathematical problems and the corresponding solutions in English.
 
For the code LoRA module: we use the Magicoder dataset [Magicoder](https://arxiv.org/abs/2312.02120), which consists of 186K code generation problems and the corresponding solutions in English.

## Data used for gates
We use gates to fuse different LoRA modules. We employ few-shot training and have released our training data. For more details, please refer to our GitHub.

# Results
We have released the results for LoRAs and LoRA-Flow

| **Method**            |       | **MGSM (Math)**   |         |         |         | **HumanEval (Code)**   |         |         |         |
|-----------------------|-------|-------------------------------|---------|---------|---------|----------------------------------|---------|---------|---------|
|                       |       | **Zh**  | **Ru**  | **Es**  | **Avg.**| **Zh**       | **Ru**  | **Es**  | **Avg.**|
| **Base Model**         |       | 4.4  | 3.2     | 2.4     | 3.3     | 0.0     | 0.0     | 2.4     | 0.8     |
| **Single LoRA**        | Lang  | 5.2 | 3.6     | 3.6     | 4.1     | 12.2     | 14.0    | 10.4    | 12.2    |
|                       | Task  | 26.8     | 32.8    | 41.2    | 33.6    | 18.3     | 23.2    | 21.9    | 21.1    |
| **LoRA Fusion**        | Avg   | 12.8   | 10.4    | 18.4    | 13.9    | 17.1   | 17.7    | 18.3    | 17.7    |
|                       | LoRA-Hub | 20.8   | 28.4    | 36.8    | 28.7    | 19.5   | 21.3    | 20.1    | 20.3    |
|                       | LoRA-Flow | **33.2**    | **37.6**| **42.0**| **37.6**| **20.7**  | **23.8**| **23.2**| **22.6**|

<table border="1" cellspacing="0" cellpadding="10">
    <caption>
        Evaluation results on MGSM and HumanEval. ‘Lang’ denotes the chat LoRA in the target language and ‘Task’ represents the math or code LoRA trained in English. LoRA fusion methods combine the language LoRA and the task LoRA to accomplish the new task. The best score is highlighted in bold.
    </caption>
    <thead>
        <tr>
            <th rowspan="2">Method</th>
            <th colspan="4">MGSM (Math)</th>
            <th colspan="4">HumanEval (Code)</th>
        </tr>
        <tr>
            <th>Zh</th>
            <th>Ru</th>
            <th>Es</th>
            <th>Avg.</th>
            <th>Zh</th>
            <th>Ru</th>
            <th>Es</th>
            <th>Avg.</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>Base Model</td>
            <td>4.4</td>
            <td>3.2</td>
            <td>2.4</td>
            <td>3.3</td>
            <td>0.0</td>
            <td>0.0</td>
            <td>2.4</td>
            <td>0.8</td>
        </tr>
        <tr>
            <td rowspan="2">Single LoRA</td>
            <td>5.2</td>
            <td>3.6</td>
            <td>3.6</td>
            <td>4.1</td>
            <td>12.2</td>
            <td>14.0</td>
            <td>10.4</td>
            <td>12.2</td>
        </tr>
        <tr>
            <td>26.8</td>
            <td>32.8</td>
            <td>41.2</td>
            <td>33.6</td>
            <td>18.3</td>
            <td>23.2</td>
            <td>21.9</td>
            <td>21.1</td>
        </tr>
        <tr>
            <td rowspan="3">LoRA Fusion</td>
            <td>12.8</td>
            <td>10.4</td>
            <td>18.4</td>
            <td>13.9</td>
            <td>17.1</td>
            <td>17.7</td>
            <td>18.3</td>
            <td>17.7</td>
        </tr>
        <tr>
            <td>20.8</td>
            <td>28.4</td>
            <td>36.8</td>
            <td>28.7</td>
            <td>19.5</td>
            <td>21.3</td>
            <td>20.1</td>
            <td>20.3</td>
        </tr>
        <tr>
            <td><strong>33.2</strong></td>
            <td><strong>37.6</strong></td>
            <td><strong>42.0</strong></td>
            <td><strong>37.6</strong></td>
            <td><strong>20.7</strong></td>
            <td><strong>23.8</strong></td>
            <td><strong>23.2</strong></td>
            <td><strong>22.6</strong></td>
        </tr>
    </tbody>
</table>





# Citation
```bibtex
@inproceedings{wang-etal-2024-lora-flow,
    title = "LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks",
    author = "Wang, Hanqing  and
      Ping, Bowen  and
      Wang, Shuo  and
      Han, Xu  and
      Chen, Yun  and
      Liu, Zhiyuan  and
      Sun, Maosong",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.695",
    doi = "10.18653/v1/2024.acl-long.695",
    pages = "12871--12882"
}
```
<!-- [LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks](https://aclanthology.org/2024.acl-long.695)  -->