Edit model card

llama2-7bn-xsum-cnn-adapter

This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on XSum and CNN/DM. LoRA adapter model based on LLama2 7bn. You can view all the implementation details on the GitHub project

Weights and Biases Documentation: Training and Eval

See Weights and Biases for training details.

Training procedure

  • Input source document wrapped in a prompt: "Summarize the following article:<INPUT>; Summary: <OUTPUT>"
  • Trained using cross-entropy on CausalLM task
  • Data splits consist of sequences up to 512 tokens:
    • Training n-datapoints: 115'354 XSum; 27494 CNN
    • Val n-datapoints: 3928 XSum; 1211 CNN

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.001
  • train_batch_size: 16
  • eval_batch_size: 16
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_ratio: 0.1
  • lr_scheduler_warmup_steps: 558.0
  • num_epochs: 2
  • mixed_precision_training: Native AMP

Training results

Achieves loss=2.021 on valdiation split, see W&B run (link above) for more details.

Framework versions

  • Transformers 4.35.0
  • Pytorch 2.0.1
  • Datasets 2.14.6
  • Tokenizers 0.14.1
Downloads last month
5
Safetensors
Model size
67M params
Tensor type
F32
·
Inference API
Unable to determine this model’s pipeline type. Check the docs .

Model tree for ernlavr/llama2-7bn-xsum-cnn-lora-adapter

Adapter
(1086)
this model

Datasets used to train ernlavr/llama2-7bn-xsum-cnn-lora-adapter