exp2-led-risalah_data_v2

This model is a fine-tuned version of silmi224/finetune-led-35000 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.6223
• Rouge1: 20.4859
• Rouge2: 10.2651
• Rougel: 14.7662
• Rougelsum: 19.2553

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 8
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 300
• num_epochs: 30
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum
3.3339	1.0	10	2.8010	8.3493	2.4084	6.4284	7.9202
3.1015	2.0	20	2.5436	8.9461	2.3615	6.7822	8.3767
2.779	3.0	30	2.2976	11.5444	3.5251	8.0258	10.4456
2.5118	4.0	40	2.1282	13.3666	4.1766	9.2522	11.9858
2.3057	5.0	50	2.0147	15.021	5.5582	10.3573	14.1171
2.1541	6.0	60	1.9283	15.937	6.8169	11.0627	14.6866
2.0326	7.0	70	1.8601	14.7364	5.5533	10.3599	13.9586
1.938	8.0	80	1.8050	14.8895	6.0535	9.9969	14.4782
1.8462	9.0	90	1.7492	14.0282	5.8353	9.232	13.2213
1.7767	10.0	100	1.7214	16.7779	7.2314	11.1359	16.1369
1.7042	11.0	110	1.6857	18.4084	8.7509	12.7906	17.8835
1.6543	12.0	120	1.6610	19.2909	8.9371	13.1256	17.6865
1.5958	13.0	130	1.6335	19.8664	9.7174	13.6907	18.8411
1.5414	14.0	140	1.6145	19.2112	9.6741	14.1273	17.7185
1.496	15.0	150	1.6234	18.8087	9.0827	13.6381	17.6146
1.4534	16.0	160	1.6035	19.4539	10.135	14.4283	18.5099
1.4177	17.0	170	1.5948	19.6367	10.405	14.0816	18.0333
1.3742	18.0	180	1.5712	18.8434	10.1431	13.7222	17.6519
1.3378	19.0	190	1.5829	18.9662	10.7079	13.9422	18.1457
1.3068	20.0	200	1.5746	20.724	11.3974	15.1529	19.8343
1.2669	21.0	210	1.5476	19.0993	9.6869	13.815	18.5096
1.2315	22.0	220	1.5606	20.4637	10.7418	14.634	19.5588
1.2005	23.0	230	1.5617	19.3271	9.8272	14.2547	18.5378
1.1649	24.0	240	1.5618	20.3699	11.3093	14.2115	19.4149
1.1344	25.0	250	1.5649	20.8124	11.3997	15.8717	20.0457
1.099	26.0	260	1.5985	19.8977	9.9926	14.1038	19.0059
1.065	27.0	270	1.5678	20.7049	10.9546	14.4462	19.5927
1.0344	28.0	280	1.6225	21.3939	11.2821	15.0261	20.3781
1.0029	29.0	290	1.5831	20.7287	11.0327	14.3893	19.9485
0.9711	30.0	300	1.6223	20.4859	10.2651	14.7662	19.2553

Framework versions

• Transformers 4.41.2
• Pytorch 2.1.2
• Datasets 2.19.2
• Tokenizers 0.19.1