distilbert-base-uncased-finetuned-pubmed-lora-trained-tabbas97

This model is a fine-tuned version of distilbert-base-uncased on the pubmed-summarization dataset. It achieves the following results on the evaluation set:

• Loss: 1.9256

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 4
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
2.1986	0.1667	500	2.0156
2.1414	0.3334	1000	1.9893
2.1247	0.5002	1500	1.9770
2.1106	0.6669	2000	1.9640
2.103	0.8336	2500	1.9548
2.0974	1.0003	3000	1.9519
2.0874	1.1671	3500	1.9506
2.0842	1.3338	4000	1.9470
2.0799	1.5005	4500	1.9406
2.0781	1.6672	5000	1.9363
2.0763	1.8339	5500	1.9371
2.0664	2.0007	6000	1.9311
2.0717	2.1674	6500	1.9277
2.0683	2.3341	7000	1.9247
2.0622	2.5008	7500	1.9290
2.0614	2.6676	8000	1.9170
2.0614	2.8343	8500	1.9239
2.0646	3.0010	9000	1.9211

Framework versions

• PEFT 0.11.1
• Transformers 4.40.2
• Pytorch 2.2.1+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1