@inproceedings{
key2023local,
title={Local Lo{RA}: Memory-Efficient Fine-Tuning of Large Language Models},
author={Oscar Key and Jean Kaddour and Pasquale Minervini},
booktitle={Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@NeurIPS 2023)},
year={2023},
url={https://openreview.net/forum?id=LHKmzWP7RN}
}
We present Local LoRA, a memory-flexible fine-tuning approach that, in principle, can fine-tune an arbitrarily large model on fixed hardware, including consumer grade GPUs. Our approach aims to decouple the size of the model and the memory required to fine-tune it by dividing the model into chunks and sequentially fine tuning each chunk. Our results show that Local LoRA closes the gap between the un-tuned model and end-to-end LoRA on math reasoning tasks.
Local LoRA: Memory-Efficient Fine-Tuning of Large Language Models
Oscar Key Jean Kaddour Pasquale Minervini