We propose a memory-efficient finetuning algorithm for large language models (LLMs)  that supports finetuning LLMs with 65B parameters in 3-bit or 4-bit precision on as little as one 48GB GPU.
Our method, modular low-rank adaptation (ModuLoRA) , integrates any user-specified weight quantizer with finetuning via low-rank adapters (LoRAs) .
Our approach relies on
a simple quantization-agnostic backward pass that adaptively materializes low-precision LLM weights from a custom black-box quantization module.
This approach enables finetuning 3-bit LLMs for the first time—leveraging state-of-the-art 3-bit OPTQ quantization often outperforms finetuning that relies on less sophisticated 4-bit and 8-bit methods.
In our experiments,ModuLoRAattains competitive performance on text classification, natural language infernece, and instruction following tasks using significantly less memory than existing approaches, and we also surpass the state-of-the-art ROUGE score on a popular summarization task.
We releaseModuLoRAtogether with a series of low-precision models—including the first family of 3-bit instruction following Alpaca LLMs—as part ofLLMTools, a user-friendly library for quantizing, running, and finetuning LLMs on consumer GPUs.