Edit model card

LoftQ Initialization

| Paper | Code | PEFT Example |

LoftQ (LoRA-fine-tuning-aware Quantization) provides a quantized backbone Q and LoRA adapters A and B, given a full-precision pre-trained weight W.

This model, LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank, is obtained from Llama-3-8B-Instruct. The backbone is under LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank and LoRA adapters are under the subfolder='loftq_init'.

Model Info

Backbone

  • Size: ~ 6 GiB
  • Loaded format: bitsandbytes nf4
  • Size loaded on GPU: ~ 6 GiB

LoRA adapters

  • rank: 64
  • lora_alpha: 16
  • target_modules: ["down_proj", "up_proj", "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj"]

Usage

Training

Here's an example of loading this model and preparing for the LoRA fine-tuning.

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

MODEL_ID = "LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank"

base_model = AutoModelForCausalLM.from_pretrained(MODEL_ID)
peft_model = PeftModel.from_pretrained(
    base_model,
    MODEL_ID,
    subfolder="loftq_init",
    is_trainable=True,
)

# Do training with peft_model ...

See the full code at our Github Repo

Citation

@article{li2023loftq,
  title={Loftq: Lora-fine-tuning-aware quantization for large language models},
  author={Li, Yixiao and Yu, Yifan and Liang, Chen and He, Pengcheng and Karampatziakis, Nikos and Chen, Weizhu and Zhao, Tuo},
  journal={arXiv preprint arXiv:2310.08659},
  year={2023}
}
Downloads last month
14
Safetensors
Model size
4.65B params
Tensor type
BF16
·
F32
·
U8
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank

Adapters
4 models

Collection including LoftQ/Meta-Llama-3-8B-Instruct-4bit-64rank