malayalam_combined_

This model is a fine-tuned version of facebook/w2v-bert-2.0 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.5025
• Wer: 0.4256

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 50
• num_epochs: 25
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer
0.8238	0.2031	500	0.8281	0.6745
0.7415	0.4063	1000	0.7477	0.6446
0.6913	0.6094	1500	0.6962	0.6072
0.6401	0.8125	2000	0.6981	0.5929
0.5864	1.0156	2500	0.6809	0.5712
0.5843	1.2188	3000	0.6125	0.5691
0.5547	1.4219	3500	0.6110	0.5616
0.5657	1.6250	4000	0.5882	0.5464
0.5809	1.8282	4500	0.5776	0.5481
0.5464	2.0313	5000	0.5689	0.5278
0.4974	2.2344	5500	0.5926	0.5428
0.5012	2.4375	6000	0.5622	0.5384
0.5162	2.6407	6500	0.5697	0.5179
0.5006	2.8438	7000	0.5357	0.5375
0.4661	3.0469	7500	0.5255	0.5255
0.4658	3.2501	8000	0.5182	0.5002
0.4716	3.4532	8500	0.5176	0.5044
0.4658	3.6563	9000	0.5139	0.5061
0.5031	3.8594	9500	0.5114	0.5068
0.4482	4.0626	10000	0.5331	0.5101
0.4678	4.2657	10500	0.5165	0.5126
0.4353	4.4688	11000	0.5292	0.5112
0.4711	4.6719	11500	0.5178	0.4979
0.4574	4.8751	12000	0.5215	0.5100
0.4246	5.0782	12500	0.5190	0.4938
0.4164	5.2813	13000	0.5504	0.4898
0.4181	5.4845	13500	0.5045	0.4979
0.4279	5.6876	14000	0.5118	0.4932
0.4244	5.8907	14500	0.4970	0.4842
0.4038	6.0938	15000	0.5013	0.4776
0.4179	6.2970	15500	0.5061	0.4762
0.3812	6.5001	16000	0.4987	0.4689
0.4217	6.7032	16500	0.4986	0.4807
0.3989	6.9064	17000	0.4905	0.4709
0.3741	7.1095	17500	0.4842	0.4700
0.3743	7.3126	18000	0.4869	0.4734
0.3785	7.5157	18500	0.4692	0.4690
0.3759	7.7189	19000	0.4691	0.4646
0.3809	7.9220	19500	0.4736	0.4720
0.3499	8.1251	20000	0.4787	0.4691
0.3523	8.3283	20500	0.4689	0.4680
0.3551	8.5314	21000	0.4792	0.4567
0.3672	8.7345	21500	0.4760	0.4652
0.3554	8.9376	22000	0.4649	0.4648
0.3182	9.1408	22500	0.4853	0.4565
0.3412	9.3439	23000	0.4958	0.4616
0.3494	9.5470	23500	0.4971	0.4527
0.3426	9.7502	24000	0.4959	0.4554
0.3365	9.9533	24500	0.4659	0.4582
0.3179	10.1564	25000	0.4807	0.4445
0.3361	10.3595	25500	0.4700	0.4535
0.3234	10.5627	26000	0.4562	0.4542
0.3296	10.7658	26500	0.4682	0.4452
0.3148	10.9689	27000	0.4716	0.4521
0.3112	11.1720	27500	0.4537	0.4473
0.3246	11.3752	28000	0.4594	0.4444
0.3062	11.5783	28500	0.4544	0.4445
0.2979	11.7814	29000	0.4531	0.4516
0.3108	11.9846	29500	0.4514	0.4428
0.2876	12.1877	30000	0.4598	0.4402
0.2911	12.3908	30500	0.4554	0.4426
0.2963	12.5939	31000	0.4641	0.4483
0.296	12.7971	31500	0.4575	0.4394
0.2777	13.0002	32000	0.4586	0.4444
0.2782	13.2033	32500	0.4498	0.4461
0.2695	13.4065	33000	0.4696	0.4450
0.286	13.6096	33500	0.4630	0.4383
0.279	13.8127	34000	0.4618	0.4401
0.2584	14.0158	34500	0.4526	0.4356
0.267	14.2190	35000	0.4726	0.4297
0.2667	14.4221	35500	0.4572	0.4308
0.2592	14.6252	36000	0.4795	0.4325
0.2592	14.8284	36500	0.4528	0.4303
0.2644	15.0315	37000	0.4604	0.4306
0.2312	15.2346	37500	0.4632	0.4367
0.2408	15.4377	38000	0.4670	0.4324
0.2489	15.6409	38500	0.4580	0.4253
0.2652	15.8440	39000	0.4581	0.4375
0.2367	16.0471	39500	0.4770	0.4213
0.2366	16.2503	40000	0.4751	0.4243
0.2267	16.4534	40500	0.4622	0.4282
0.2461	16.6565	41000	0.4671	0.4249
0.2326	16.8596	41500	0.4736	0.4293
0.2121	17.0628	42000	0.4905	0.4300
0.222	17.2659	42500	0.4782	0.4261
0.2202	17.4690	43000	0.4670	0.4250
0.2141	17.6722	43500	0.4688	0.4259
0.2231	17.8753	44000	0.4718	0.4254
0.2144	18.0784	44500	0.5025	0.4256

Framework versions

• Transformers 4.43.0.dev0
• Pytorch 1.14.0a0+44dac51
• Datasets 2.16.1
• Tokenizers 0.19.1