nyrkr-joker-llama
New Yorker cartoon description and caption -> attempt at a joke explanation
Technical details:
- Based on LLaMa2-7b-hf (version 2, 7B params)
- Used QLoRA to fine-tune on 1.2k rows of New Yorker caption contest
- Merged LLaMa2 with the adapter weights (from checkpoint step=160, epoch=2.7)
Prompt options
The original paper, Figure 10 uses this format for joke explanations:
`In this task, you will see a description of an uncanny situation. Then, you will see a joke that was written about the situation. Explain how the joke relates to the situation and why it is funny. ###
{few-shot examples separated by ###, newline after "explanation of the caption:"} This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller. caption: Can I interest you in opening an offshore account? explanation of the caption: `
In training, I used just the individual example:
This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller. caption: Can I interest you in opening an offshore account? explanation of the caption:\n
In inference, I had some better results with a more natural prompt (no newline or space at end)
This scene takes place in the following location: a bank. Three people are standing in line at the bank. The bank teller is a traditional pirate with a hook hand, eye patch, and a parrot. The scene includes: Piracy, Bank teller. caption: Can I interest you in opening an offshore account? the caption is funny because
Training script
Trained on a V100
git clone https://github.com/artidoro/qlora
cd qlora
pip3 install -r requirements.txt --quiet
! cd qlora && python qlora.py \
--model_name_or_path ../llama-2-7b-hf \
--output_dir ../thatsthejoke \
--logging_steps 20 \
--save_strategy steps \
--data_seed 42 \
--save_steps 80 \
--save_total_limit 10 \
--evaluation_strategy steps \
--max_new_tokens 64 \
--dataloader_num_workers 1 \
--group_by_length \
--logging_strategy steps \
--remove_unused_columns False \
--do_train \
--lora_r 64 \
--lora_alpha 16 \
--lora_modules all \
--double_quant \
--quant_type nf4 \
--bits 4 \
--warmup_ratio 0.03 \
--lr_scheduler_type constant \
--gradient_checkpointing \
--dataset /content/nycaptions.jsonl \
--dataset_format 'self-instruct' \
--source_max_len 16 \
--target_max_len 512 \
--per_device_train_batch_size 1 \
--gradient_accumulation_steps 16 \
--max_steps 250 \
--eval_steps 187 \
--learning_rate 0.0002 \
--adam_beta2 0.999 \
--max_grad_norm 0.3 \
--lora_dropout 0.1 \
--weight_decay 0.0 \
--seed 0
- Downloads last month
- 8