flan-t5-small-tldr-50k

This model is a fine-tuned version of google/flan-t5-small on the Reddit TL;DR dataset (https://zenodo.org/record/1168855#.ZB8P-iFByUk). It achieves the following results on the evaluation set:

• Gen Len: 16.4422
• Loss: 3.2423
• Rouge1: 14.7049
• Rouge2: 3.2396
• Rougel: 12.5104
• Rougelsum: 12.9681

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Gen Len	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum
3.5507	1.0	5625	16.1424	3.2752	14.2302	2.9853	12.1734	12.5894
3.4842	2.0	11250	16.1126	3.2569	14.3966	3.0939	12.2437	12.6705
3.4288	3.0	16875	16.39	3.2481	14.6879	3.2647	12.5199	12.9681
3.4176	4.0	22500	16.2948	3.2432	14.7198	3.2693	12.5436	12.9885
3.4033	5.0	28125	16.4422	3.2423	14.7049	3.2396	12.5104	12.9681

Framework versions

• Transformers 4.27.3
• Pytorch 1.13.1
• Datasets 2.10.1
• Tokenizers 0.13.2