t5-small-mlm-pubmed-35

This model is a fine-tuned version of t5-small on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.1101
• Rouge2 Precision: 0.4758
• Rouge2 Recall: 0.3498
• Rouge2 Fmeasure: 0.3927

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 40
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge2 Precision	Rouge2 Recall	Rouge2 Fmeasure
1.8404	0.75	500	1.5005	0.4265	0.2786	0.3273
1.6858	1.51	1000	1.4216	0.4318	0.2946	0.3404
1.6071	2.26	1500	1.3777	0.4472	0.3148	0.3598
1.5551	3.02	2000	1.3360	0.4406	0.3168	0.3586
1.5116	3.77	2500	1.3128	0.4523	0.3234	0.3671
1.4837	4.52	3000	1.2937	0.4477	0.3215	0.3645
1.4513	5.28	3500	1.2766	0.4511	0.3262	0.3689
1.4336	6.03	4000	1.2626	0.4548	0.3283	0.3718
1.4149	6.79	4500	1.2449	0.4495	0.3274	0.3687
1.3977	7.54	5000	1.2349	0.4507	0.3305	0.3712
1.3763	8.3	5500	1.2239	0.4519	0.3266	0.3688
1.371	9.05	6000	1.2171	0.4546	0.3305	0.3727
1.3501	9.8	6500	1.2080	0.4575	0.3329	0.3755
1.3443	10.56	7000	1.2017	0.4576	0.3314	0.3742
1.326	11.31	7500	1.1926	0.4578	0.333	0.3757
1.3231	12.07	8000	1.1866	0.4606	0.3357	0.3782
1.3089	12.82	8500	1.1816	0.4591	0.3338	0.3765
1.3007	13.57	9000	1.1764	0.4589	0.3361	0.3777
1.2943	14.33	9500	1.1717	0.4641	0.3382	0.3811
1.2854	15.08	10000	1.1655	0.4617	0.3378	0.38
1.2777	15.84	10500	1.1612	0.464	0.3401	0.3823
1.2684	16.59	11000	1.1581	0.4608	0.3367	0.3789
1.2612	17.35	11500	1.1554	0.4623	0.3402	0.3818
1.2625	18.1	12000	1.1497	0.4613	0.3381	0.3802
1.2529	18.85	12500	1.1465	0.4671	0.3419	0.3848
1.2461	19.61	13000	1.1431	0.4646	0.3399	0.3824
1.2415	20.36	13500	1.1419	0.4659	0.341	0.3835
1.2375	21.12	14000	1.1377	0.4693	0.3447	0.3873
1.2315	21.87	14500	1.1353	0.4672	0.3433	0.3855
1.2263	22.62	15000	1.1333	0.467	0.3433	0.3854
1.2214	23.38	15500	1.1305	0.4682	0.3446	0.3869
1.2202	24.13	16000	1.1291	0.4703	0.3465	0.3888
1.2155	24.89	16500	1.1270	0.472	0.348	0.3903
1.2064	25.64	17000	1.1261	0.4724	0.3479	0.3905
1.2173	26.4	17500	1.1236	0.4734	0.3485	0.3912
1.1994	27.15	18000	1.1220	0.4739	0.3486	0.3915
1.2018	27.9	18500	1.1217	0.4747	0.3489	0.3921
1.2045	28.66	19000	1.1194	0.4735	0.3488	0.3916
1.1949	29.41	19500	1.1182	0.4732	0.3484	0.3911
1.19	30.17	20000	1.1166	0.4724	0.3479	0.3904
1.1932	30.92	20500	1.1164	0.4753	0.3494	0.3924
1.1952	31.67	21000	1.1147	0.4733	0.3485	0.3911
1.1922	32.43	21500	1.1146	0.475	0.3494	0.3923
1.1889	33.18	22000	1.1132	0.4765	0.3499	0.3933
1.1836	33.94	22500	1.1131	0.4768	0.351	0.3939
1.191	34.69	23000	1.1127	0.4755	0.3495	0.3926
1.1811	35.44	23500	1.1113	0.4748	0.349	0.3919
1.1864	36.2	24000	1.1107	0.4751	0.3494	0.3921
1.1789	36.95	24500	1.1103	0.4756	0.3499	0.3927
1.1819	37.71	25000	1.1101	0.4758	0.35	0.3932
1.1862	38.46	25500	1.1099	0.4755	0.3497	0.3926
1.1764	39.22	26000	1.1101	0.4759	0.3498	0.3928
1.1819	39.97	26500	1.1101	0.4758	0.3498	0.3927

Framework versions

• Transformers 4.12.5
• Pytorch 1.10.0+cu111
• Datasets 1.15.1
• Tokenizers 0.10.3