Update README.md

README.md

@@ -19,21 +19,21 @@ We released all of our checkpoints used in [LoRA-Flow](https://aclanthology.org/
 # Summary
 >  In this repo, we release LoRA modules and the gate of 7B models trained in our paper in HuggingFace format.
 # Introduction
-LoRA-Flow provides an efficient way to fuse different LoRA modules which can outperform existing methods significantly. The following picture shows our proposed method, we use layer-wise fusion gates to facilitate dynamic LoRA fusion, which project input hidden states of each layer into fusion weights. For more details, please refer to our paper.
+LoRA-Flow provides an efficient way to fuse different LoRA modules. The following picture shows our proposed method, we use layer-wise fusion gates to facilitate dynamic LoRA fusion, which project input hidden states of each layer into fusion weights. For more details, please refer to our paper.
 ![1.jpg](https://cdn-uploads.huggingface.co/production/uploads/64d99f6cd7e30889c6c477b4/ifiu1FTHilrmUkD4FKkgV.jpeg)
-# Training Details
-## LoRA modules Training
-For language LoRA modules: we use the 52K training samples respectively which from [Okapi](https://aclanthology.org/2023.emnlp-demo.28).
+# Training Data
+## Data used for LoRA modules
+For the language LoRA modules: we use the 52K training samples from [Okapi](https://aclanthology.org/2023.emnlp-demo.28) for each language, respectively.
 
-For math LoRA module: the training data for English math LoRA is constructed by [Metamath](https://arxiv.org/abs/2309.12284), which is comprised of 395K mathematical problems in English.
+For the math LoRA module: we use [Metamath](https://arxiv.org/abs/2309.12284) that is comprised of 395K mathematical problems and the corresponding solutions in English.
  
-For code LoRA module: we train the English code LoRA with the Magicoder dataset [Magicoder](https://arxiv.org/abs/2312.02120), which consists of 186K code generation problems in English.
+For the code LoRA module: we use the Magicoder dataset [Magicoder](https://arxiv.org/abs/2312.02120), which consists of 186K code generation problems and the corresponding solutions in English.
 
-## Gate Training
-We use gates to fuse different LoRA modules. We employ few-shot training and have released our training data for further details please refer to our GitHub.
+## Data used for gates
+We use gates to fuse different LoRA modules. We employ few-shot training and have released our training data. For more details, please refer to our GitHub.
 
-# Result
-We have released the result for LoRAs and LoRA-Flow
+# Results
+We have released the results for LoRAs and LoRA-Flow
 
 | **Method**            |       | **MGSM (Math)**   |         |         |         | **HumanEval (Code)**   |         |         |         |
 |-----------------------|-------|-------------------------------|---------|---------|---------|----------------------------------|---------|---------|---------|
@@ -47,7 +47,6 @@ We have released the result for LoRAs and LoRA-Flow
 
 
 # Citation
-if you find our repo is helpful, please cite the following 
 ```bibtex
 @inproceedings{wang-etal-2024-lora-flow,
     title = "LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks",
@@ -58,9 +57,6 @@ if you find our repo is helpful, please cite the following
       Chen, Yun  and
       Liu, Zhiyuan  and
       Sun, Maosong",
-    editor = "Ku, Lun-Wei  and
-      Martins, Andre  and
-      Srikumar, Vivek",
     booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
     month = aug,
     year = "2024",
@@ -68,8 +64,7 @@ if you find our repo is helpful, please cite the following
     publisher = "Association for Computational Linguistics",
     url = "https://aclanthology.org/2024.acl-long.695",
     doi = "10.18653/v1/2024.acl-long.695",
-    pages = "12871--12882",
-    abstract = "LoRA employs lightweight modules to customize large language models (LLMs) for each downstream task or domain, where different learned additional modules represent diverse skills. Combining existing LoRAs to address new tasks can enhance the reusability of learned LoRAs, particularly beneficial for tasks with limited annotated data. Most prior works on LoRA combination primarily rely on task-level weights for each involved LoRA, making different examples and tokens share the same LoRA weights. However, in generative tasks, different tokens may necessitate diverse skills to manage. Taking the Chinese math task as an example, understanding the problem description may depend more on the Chinese LoRA, while the calculation part may rely more on the math LoRA. To this end, we propose LoRA-Flow, which utilizes dynamic weights to adjust the impact of different LoRAs. The weights at each step are determined by a fusion gate with extremely few parameters, which can be learned with only 200 training examples. Experiments across six generative tasks demonstrate that our method consistently outperforms baselines with task-level fusion weights. This underscores the necessity of introducing dynamic fusion weights for LoRA combination.",
+    pages = "12871--12882"
 }
 ```
 <!-- [LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks](https://aclanthology.org/2024.acl-long.695)  -->