|
--- |
|
license: mit |
|
datasets: |
|
- yahma/alpaca-cleaned |
|
--- |
|
This repo contains a low-rank adapter for LLaMA-13b fit on the Cleaned Alpaca dataset containing the new GPT-4 data. |
|
|
|
This version of the weights was trained with the following hyperparameters: |
|
|
|
Cleaned dataset: Snapshot April 9, 2023 |
|
Epochs: 4 |
|
Validation set size: 1500 |
|
Batch size: 128 |
|
Micro batch size: 4 |
|
Cutoff length: 512 |
|
Learning rate: 3e-4 |
|
Lora r: 16 |
|
Lora target modules: q_proj, k_proj, v_proj, o_proj |
|
|
|
That is: |
|
|
|
python finetune.py \ |
|
--base_model='yahma/llama-13b-hf' \ |
|
--data_path 'yahma/alpaca-cleaned' \ |
|
--num_epochs=4 \ |
|
--cutoff_len=512 \ |
|
--output_dir='./lora-alpaca' \ |
|
--lora_target_modules='[q_proj,k_proj, v_proj, o_proj]' \ |
|
--lora_r=16 \ |
|
--val_set_size 1500 \ |
|
--micro_batch_size=4 |
