Edit model card

Llama-3.2-1B-Summarization-QLoRa

This model is a fine-tuned version of meta-llama/Llama-3.2-1B on the scitldr dataset. It achieves the following results on the evaluation set:

  • Loss: 2.5899

Model description

Fine-tuned (QLoRa) Version of Meta-llama/Llama-3.2-1B for Summarization of scientific documents.

Intended uses & limitations

Summarization

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • optimizer: Use paged_adamw_32bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 2
  • num_epochs: 2
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
2.4993 0.2008 200 2.5715
2.4748 0.4016 400 2.5674
2.4744 0.6024 600 2.5674
2.4646 0.8032 800 2.5558
2.4637 1.0040 1000 2.5539
2.1281 1.2048 1200 2.5904
2.1157 1.4056 1400 2.5928
2.0962 1.6064 1600 2.5855
2.0721 1.8072 1800 2.5899

Framework versions

  • PEFT 0.13.2
  • Transformers 4.46.2
  • Pytorch 2.5.1+cu121
  • Datasets 3.1.0
  • Tokenizers 0.20.3
Downloads last month
34
Inference Examples
Inference API (serverless) does not yet support peft models for this pipeline type.

Model tree for pkbiswas/Llama-3.2-1B-Summarization-QLoRa

Adapter
(69)
this model

Dataset used to train pkbiswas/Llama-3.2-1B-Summarization-QLoRa