longt5_xl_gov_memsum_bp_15

This model is a fine-tuned version of /exports/eddie/scratch/s1970716/models/summarization/longt5_xl_gov_memsum_bp_10/checkpoint-1360 on the learn3r/gov_report_memsum_bp dataset. It achieves the following results on the evaluation set:

• Loss: 3.2731
• Rouge1: 35.319
• Rouge2: 11.8288
• Rougel: 16.3777
• Rougelsum: 33.5357
• Gen Len: 1942.3155

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.001
• train_batch_size: 1
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 64
• total_train_batch_size: 64
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• num_epochs: 5.0

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
0.1366	1.0	272	3.2731	35.319	11.8288	16.3777	33.5357	1942.3155
0.1183	2.0	545	3.3957	37.5265	12.4932	16.743	35.7285	1867.3145
0.1072	3.0	818	3.4308	41.1487	13.4035	17.5783	39.1233	1561.0853
0.0909	4.0	1091	3.6078	41.2814	13.3137	17.9878	39.2664	1429.4604
0.0834	4.99	1360	3.8803	42.0328	13.9186	17.8203	39.9705	1559.3237

Framework versions

• Transformers 4.34.1
• Pytorch 2.1.0+cu121
• Datasets 2.14.5
• Tokenizers 0.14.1