Edit model card

Llama-3.2-1B-Summarization-QLoRa

This model is a fine-tuned version of meta-llama/Llama-3.2-1B on the scitldr dataset. It achieves the following results on the evaluation set:

  • Loss: 2.5948

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 2
  • num_epochs: 2
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
2.5121 0.2008 200 2.5718
2.485 0.4016 400 2.5652
2.4865 0.6024 600 2.5660
2.4767 0.8032 800 2.5562
2.4744 1.0040 1000 2.5508
2.137 1.2048 1200 2.5899
2.1268 1.4056 1400 2.5914
2.108 1.6064 1600 2.5874
2.0804 1.8072 1800 2.5948

Framework versions

  • PEFT 0.13.2
  • Transformers 4.44.2
  • Pytorch 2.5.0+cu121
  • Datasets 3.0.2
  • Tokenizers 0.19.1
Downloads last month
13
Inference API
Unable to determine this model’s pipeline type. Check the docs .

Model tree for pkbiswas/Llama-3.2-1B-Summarization-QLoRa

Adapter
(40)
this model

Dataset used to train pkbiswas/Llama-3.2-1B-Summarization-QLoRa