w2v-bert-2.0-lg-cv-10hr-v2

This model is a fine-tuned version of facebook/w2v-bert-2.0 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: inf
• Model Preparation Time: 0.0157
• Wer: 0.3092
• Cer: 0.0767

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 100
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Model Preparation Time	Wer	Cer
2.608	1.0	191	0.3941	0.0157	0.4731	0.0904
0.3253	2.0	382	0.3308	0.0157	0.3739	0.0706
0.2561	3.0	573	0.3141	0.0157	0.3981	0.0728
0.2177	4.0	764	0.2993	0.0157	0.3228	0.0625
0.1819	5.0	955	0.2921	0.0157	0.3318	0.0619
0.1529	6.0	1146	0.2879	0.0157	0.3084	0.0579
0.131	7.0	1337	0.2947	0.0157	0.3132	0.0583
0.1166	8.0	1528	0.2913	0.0157	0.3095	0.0588
0.1034	9.0	1719	0.2852	0.0157	0.2911	0.0538
0.091	10.0	1910	0.3178	0.0157	0.2966	0.0565
0.0812	11.0	2101	0.3147	0.0157	0.2928	0.0539
0.0743	12.0	2292	0.3236	0.0157	0.3075	0.0576
0.0703	13.0	2483	0.3004	0.0157	0.2872	0.0544
0.0659	14.0	2674	0.3491	0.0157	0.3008	0.0573
0.0621	15.0	2865	0.3260	0.0157	0.2910	0.0554
0.0572	16.0	3056	0.3438	0.0157	0.3050	0.0582
0.0586	17.0	3247	0.3735	0.0157	0.2983	0.0558
0.056	18.0	3438	0.3911	0.0157	0.2923	0.0563
0.0656	19.0	3629	0.3602	0.0157	0.3159	0.0638
0.0664	20.0	3820	0.3857	0.0157	0.3254	0.0626
0.0786	21.0	4011	0.3534	0.0157	0.3649	0.0723
0.1442	22.0	4202	0.3541	0.0157	0.3444	0.0669
0.1176	23.0	4393	0.3509	0.0157	0.3333	0.0649

Framework versions

• Transformers 4.44.2
• Pytorch 2.1.0+cu118
• Datasets 2.20.0
• Tokenizers 0.19.1