roberta-base_adalora_lr5e-05_bs4_epoch20_wd0.01

This model is a fine-tuned version of roberta-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.9446

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss
20.9059	1.0	157	22.9422
19.6988	2.0	314	20.5387
16.3079	3.0	471	8.3490
6.1827	4.0	628	5.2791
5.1076	5.0	785	4.0504
3.821	6.0	942	3.1710
3.3796	7.0	1099	2.5361
2.6715	8.0	1256	2.0947
2.2	9.0	1413	1.7770
2.0171	10.0	1570	1.5355
1.7243	11.0	1727	1.3742
1.5945	12.0	1884	1.2590
1.4534	13.0	2041	1.1651
1.375	14.0	2198	1.0983
1.3396	15.0	2355	1.0479
1.2208	16.0	2512	1.0039
1.2219	17.0	2669	0.9751
1.2019	18.0	2826	0.9588
1.1798	19.0	2983	0.9481
1.1494	20.0	3140	0.9446

Framework versions

• PEFT 0.7.1
• Transformers 4.36.2
• Pytorch 2.0.1
• Datasets 2.16.1
• Tokenizers 0.15.0