llama2-7bn-xsum-cnn-adapter
This model is a fine-tuned version of meta-llama/Llama-2-7b-hf on XSum and CNN/DM. LoRA adapter model based on LLama2 7bn. You can view all the implementation details on the GitHub project
Weights and Biases Documentation: Training and Eval
See Weights and Biases for training details.
Training procedure
- Input source document wrapped in a prompt: "Summarize the following article:<INPUT>; Summary: <OUTPUT>"
- Trained using cross-entropy on CausalLM task
- Data splits consist of sequences up to 512 tokens:
- Training n-datapoints: 115'354 XSum; 27494 CNN
- Val n-datapoints: 3928 XSum; 1211 CNN
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.001
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- lr_scheduler_warmup_steps: 558.0
- num_epochs: 2
- mixed_precision_training: Native AMP
Training results
Achieves loss=2.021 on valdiation split, see W&B run (link above) for more details.
Framework versions
- Transformers 4.35.0
- Pytorch 2.0.1
- Datasets 2.14.6
- Tokenizers 0.14.1
- Downloads last month
- 5
Model tree for ernlavr/llama2-7bn-xsum-cnn-lora-adapter
Base model
meta-llama/Llama-2-7b-hf