metadata

license: mit

LORA-Flow

LoRAs and fusion gates for our paper

Paper • Github

We released all of our checkpoints used in LoRA-Flow which has been accepted to ACL 2024 main conference.

Summary

In this repo, we release LoRA modules and the gate of 7B models trained in our paper in HuggingFace format.

Introduction

LoRA-Flow provides an efficient way to fuse different LoRA modules. The following picture shows our proposed method, we use layer-wise fusion gates to facilitate dynamic LoRA fusion, which project input hidden states of each layer into fusion weights. LoRA-flow can be applied into Llama-7b backbone . For more details, please refer to our paper.

Training Data

Data used for LoRA modules

For the language LoRA modules: we use the 52K training samples from Okapi for each language, respectively.

For the math LoRA module: we use Metamath that is comprised of 395K mathematical problems and the corresponding solutions in English.

For the code LoRA module: we use the Magicoder dataset Magicoder, which consists of 186K code generation problems and the corresponding solutions in English.

Data used for gates

We use gates to fuse different LoRA modules. We employ few-shot training and have released our training data. For more details, please refer to our GitHub.

Results

We have released the results for LoRAs and LoRA-Flow

Method		MGSM (Math)				HumanEval (Code)
		Zh	Ru	Es	Avg.	Zh	Ru	Es	Avg.
Base Model		4.4	3.2	2.4	3.3	0.0	0.0	2.4	0.8
Single LoRA	Lang	5.2	3.6	3.6	4.1	12.2	14.0	10.4	12.2
	Task	26.8	32.8	41.2	33.6	18.3	23.2	21.9	21.1
LoRA Fusion	Avg	12.8	10.4	18.4	13.9	17.1	17.7	18.3	17.7
	LoRA-Hub	20.8	28.4	36.8	28.7	19.5	21.3	20.1	20.3
	LoRA-Flow	33.2	37.6	42.0	37.6	20.7	23.8	23.2	22.6

Evaluation results on MGSM and HumanEval. ‘Lang’ denotes the chat LoRA in the target language and ‘Task’ represents the math or code LoRA trained in English. LoRA fusion methods combine the language LoRA and the task LoRA to accomplish the new task. The best score is highlighted in bold.
Method	MGSM (Math)				HumanEval (Code)
Method	Zh	Ru	Es	Avg.	Zh	Ru	Es	Avg.
Base Model	4.4	3.2	2.4	3.3	0.0	0.0	2.4	0.8
Single LoRA	5.2	3.6	3.6	4.1	12.2	14.0	10.4	12.2
Single LoRA	26.8	32.8	41.2	33.6	18.3	23.2	21.9	21.1
LoRA Fusion	12.8	10.4	18.4	13.9	17.1	17.7	18.3	17.7
	20.8	28.4	36.8	28.7	19.5	21.3	20.1	20.3
	33.2	37.6	42.0	37.6	20.7	23.8	23.2	22.6

Citation

@inproceedings{wang-etal-2024-lora-flow,
    title = "LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks",
    author = "Wang, Hanqing  and
      Ping, Bowen  and
      Wang, Shuo  and
      Han, Xu  and
      Chen, Yun  and
      Liu, Zhiyuan  and
      Sun, Maosong",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.695",
    doi = "10.18653/v1/2024.acl-long.695",
    pages = "12871--12882"
}