File size: 6,512 Bytes
762668d
 
 
4385b31
 
 
 
 
 
 
 
 
 
 
 
 
 
762668d
 
b026360
4385b31
3c23448
762668d
4827ef5
 
 
b026360
4827ef5
b026360
4827ef5
b026360
4827ef5
 
b026360
4827ef5
 
b058e42
 
6a3a2f7
 
 
68a4d05
5f1ae17
 
6a3a2f7
 
a48048f
 
 
 
 
 
 
 
6a3a2f7
 
 
 
68a4d05
a48048f
 
 
 
 
 
 
 
6a3a2f7
 
 
a48048f
 
 
 
 
 
 
 
6a3a2f7
 
a48048f
 
 
 
 
 
 
 
6a3a2f7
 
 
a48048f
 
 
 
 
 
 
 
6a3a2f7
 
a48048f
 
 
 
 
 
 
 
6a3a2f7
 
a48048f
 
 
 
 
 
 
 
6a3a2f7
 
 
 
 
b058e42
68a4d05
5f1ae17
a48048f
 
762668d
4385b31
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4827ef5
4385b31
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
---
license: mit
---
<div align="center">

<!-- <img src="title.png" alt="LoRA-Flow" width="200"> -->
<!-- ***LORA-Flow*** -->
<b><i style="font-size: 24px;">LORA-Flow</i></b>

LoRAs and fusion gates for our paper

<p align="center">
 <a href="https://aclanthology.org/2024.acl-long.695/">Paper</a><a href="https://github.com/pingbowen23/LoRA-Flow"> Github</a>
</p>
</div>

We released all of our checkpoints used in [LoRA-Flow](https://aclanthology.org/2024.acl-long.695.pdf) which has been accepted to ACL 2024 main conference.
# Summary
>  In this repo, we release LoRA modules and the gate of 7B models trained in our paper in HuggingFace format.
# Introduction
LoRA-Flow provides an efficient way to fuse different LoRA modules. The following picture shows our proposed method, we use layer-wise fusion gates to facilitate dynamic LoRA fusion, which project input hidden states of each layer into fusion weights. LoRA-flow can be applied into [Llama-7b backbone](https://huggingface.co/meta-llama/Llama-2-7b) . For more details, please refer to our paper. 
![1.jpg](https://cdn-uploads.huggingface.co/production/uploads/64d99f6cd7e30889c6c477b4/ifiu1FTHilrmUkD4FKkgV.jpeg)
# Training Data
## Data used for LoRA modules
For the language LoRA modules: we use the 52K training samples from [Okapi](https://aclanthology.org/2023.emnlp-demo.28) for each language, respectively.

For the math LoRA module: we use [Metamath](https://arxiv.org/abs/2309.12284) that is comprised of 395K mathematical problems and the corresponding solutions in English.
 
For the code LoRA module: we use the Magicoder dataset [Magicoder](https://arxiv.org/abs/2312.02120), which consists of 186K code generation problems and the corresponding solutions in English.

## Data used for gates
We use gates to fuse different LoRA modules. We employ few-shot training and have released our training data. For more details, please refer to our GitHub.

# Results
We have released the results for LoRAs and LoRA-Flow


<table border="1" cellspacing="0" cellpadding="10">
    <thead>
        <tr>
            <th rowspan="2">Method</th>
            <th colspan="4">MGSM (Math)</th>
            <th colspan="4">HumanEval (Code)</th>
        </tr>
        <tr>
            <th style="text-align: right;">Zh</th>
            <th style="text-align: right;">Ru</th>
            <th style="text-align: right;">Es</th>
            <th style="text-align: right;">Avg.</th>
            <th style="text-align: right;">Zh</th>
            <th style="text-align: right;">Ru</th>
            <th style="text-align: right;">Es</th>
            <th style="text-align: right;">Avg.</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td>Base Model</td>
            <td style="text-align: right;">4.4</td>
            <td style="text-align: right;">3.2</td>
            <td style="text-align: right;">2.4</td>
            <td style="text-align: right;">3.3</td>
            <td style="text-align: right;">0.0</td>
            <td style="text-align: right;">0.0</td>
            <td style="text-align: right;">2.4</td>
            <td style="text-align: right;">0.8</td>
        </tr>
        <tr>
            <td rowspan="2">Single LoRA</td>
            <td style="text-align: right;">5.2</td>
            <td style="text-align: right;">3.6</td>
            <td style="text-align: right;">3.6</td>
            <td style="text-align: right;">4.1</td>
            <td style="text-align: right;">12.2</td>
            <td style="text-align: right;">14.0</td>
            <td style="text-align: right;">10.4</td>
            <td style="text-align: right;">12.2</td>
        </tr>
        <tr>
            <td style="text-align: right;">26.8</td>
            <td style="text-align: right;">32.8</td>
            <td style="text-align: right;">41.2</td>
            <td style="text-align: right;">33.6</td>
            <td style="text-align: right;">18.3</td>
            <td style="text-align: right;">23.2</td>
            <td style="text-align: right;">21.9</td>
            <td style="text-align: right;">21.1</td>
        </tr>
        <tr>
            <td rowspan="3">LoRA Fusion</td>
            <td style="text-align: right;">12.8</td>
            <td style="text-align: right;">10.4</td>
            <td style="text-align: right;">18.4</td>
            <td style="text-align: right;">13.9</td>
            <td style="text-align: right;">17.1</td>
            <td style="text-align: right;">17.7</td>
            <td style="text-align: right;">18.3</td>
            <td style="text-align: right;">17.7</td>
        </tr>
        <tr>
            <td style="text-align: right;">20.8</td>
            <td style="text-align: right;">28.4</td>
            <td style="text-align: right;">36.8</td>
            <td style="text-align: right;">28.7</td>
            <td style="text-align: right;">19.5</td>
            <td style="text-align: right;">21.3</td>
            <td style="text-align: right;">20.1</td>
            <td style="text-align: right;">20.3</td>
        </tr>
        <tr>
            <td style="text-align: right;"><strong>33.2</strong></td>
            <td style="text-align: right;"><strong>37.6</strong></td>
            <td style="text-align: right;"><strong>42.0</strong></td>
            <td style="text-align: right;"><strong>37.6</strong></td>
            <td style="text-align: right;"><strong>20.7</strong></td>
            <td style="text-align: right;"><strong>23.8</strong></td>
            <td style="text-align: right;"><strong>23.2</strong></td>
            <td style="text-align: right;"><strong>22.6</strong></td>
        </tr>
    </tbody>
</table>







# Citation
```bibtex
@inproceedings{wang-etal-2024-lora-flow,
    title = "LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks",
    author = "Wang, Hanqing  and
      Ping, Bowen  and
      Wang, Shuo  and
      Han, Xu  and
      Chen, Yun  and
      Liu, Zhiyuan  and
      Sun, Maosong",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.695",
    doi = "10.18653/v1/2024.acl-long.695",
    pages = "12871--12882"
}
```
<!-- [LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks](https://aclanthology.org/2024.acl-long.695)  -->