sulaimank/w2v-bert-grain-lg-v2

w

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the Grain dataset. It achieves the following results on the evaluation set:

• Loss: 0.0469
• Wer: 0.0299
• Cer: 0.0077

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Cer
0.2995	1.0	1164	0.1521	0.1390	0.0283
0.1049	2.0	2328	0.0931	0.0946	0.0189
0.0719	3.0	3492	0.0861	0.0902	0.0183
0.0546	4.0	4656	0.0788	0.0704	0.0166
0.0447	5.0	5820	0.0609	0.0627	0.0135
0.0374	6.0	6984	0.0744	0.0618	0.0141
0.0338	7.0	8148	0.0673	0.0535	0.0137
0.029	8.0	9312	0.0770	0.0540	0.0128
0.0278	9.0	10476	0.0565	0.0482	0.0116
0.0227	10.0	11640	0.0516	0.0500	0.0115
0.0211	11.0	12804	0.0457	0.0392	0.0096
0.0207	12.0	13968	0.0527	0.0452	0.0098
0.0179	13.0	15132	0.0463	0.0370	0.0089
0.017	14.0	16296	0.0530	0.0452	0.0109
0.0167	15.0	17460	0.0447	0.0360	0.0091
0.0141	16.0	18624	0.0529	0.0434	0.0104
0.015	17.0	19788	0.0410	0.0387	0.0090
0.0141	18.0	20952	0.0480	0.0416	0.0102
0.0136	19.0	22116	0.0472	0.0368	0.0087
0.0125	20.0	23280	0.0428	0.0380	0.0091
0.0117	21.0	24444	0.0375	0.0328	0.0081
0.0113	22.0	25608	0.0392	0.0312	0.0083
0.0093	23.0	26772	0.0554	0.0394	0.0102
0.0111	24.0	27936	0.0624	0.0452	0.0108
0.0107	25.0	29100	0.0390	0.0346	0.0076
0.0082	26.0	30264	0.0505	0.0426	0.0101
0.0087	27.0	31428	0.0430	0.0320	0.0081
0.0086	28.0	32592	0.0541	0.0398	0.0101
0.0079	29.0	33756	0.0404	0.0304	0.0070
0.0084	30.0	34920	0.0416	0.0315	0.0075
0.0084	31.0	36084	0.0495	0.0366	0.0092
0.0075	32.0	37248	0.0469	0.0299	0.0077

Framework versions

• Transformers 4.45.2
• Pytorch 2.1.0+cu118
• Datasets 3.0.1
• Tokenizers 0.20.1