cantillation/Teamim-small_WeightDecay-0.05_Augmented_Old-Data_date-23-07-2024

he-cantillation

This model is a fine-tuned version of openai/whisper-small on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.3963
• Wer: 16.4230
• Avg Precision Exact: 0.8621
• Avg Recall Exact: 0.8754
• Avg F1 Exact: 0.8679
• Avg Precision Letter Shift: 0.8906
• Avg Recall Letter Shift: 0.9042
• Avg F1 Letter Shift: 0.8966
• Avg Precision Word Level: 0.8954
• Avg Recall Word Level: 0.9066
• Avg F1 Word Level: 0.9002
• Avg Precision Word Shift: 0.9532
• Avg Recall Word Shift: 0.9639
• Avg F1 Word Shift: 0.9577
• Precision Median Exact: 0.9231
• Recall Median Exact: 0.9231
• F1 Median Exact: 0.9333
• Precision Max Exact: 1.0
• Recall Max Exact: 1.0
• F1 Max Exact: 1.0
• Precision Min Exact: 0.0
• Recall Min Exact: 0.0
• F1 Min Exact: 0.0
• Precision Min Letter Shift: 0.0
• Recall Min Letter Shift: 0.0
• F1 Min Letter Shift: 0.0
• Precision Min Word Level: 0.0
• Recall Min Word Level: 0.0
• F1 Min Word Level: 0.0
• Precision Min Word Shift: 0.6154
• Recall Min Word Shift: 0.7
• F1 Min Word Shift: 0.6667

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 8
• eval_batch_size: 32
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• training_steps: 200000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Wer	Avg Precision Exact	Avg Recall Exact	Avg F1 Exact	Avg Precision Letter Shift	Avg Recall Letter Shift	Avg F1 Letter Shift	Avg Precision Word Level	Avg Recall Word Level	Avg F1 Word Level	Avg Precision Word Shift	Avg Recall Word Shift	Avg F1 Word Shift	Precision Median Exact	Recall Median Exact	F1 Median Exact	Precision Max Exact	Recall Max Exact	F1 Max Exact	Precision Min Exact	Recall Min Exact	F1 Min Exact	Precision Min Letter Shift	Recall Min Letter Shift	F1 Min Letter Shift	Precision Min Word Level	Recall Min Word Level	F1 Min Word Level	Precision Min Word Shift	Recall Min Word Shift	F1 Min Word Shift
No log	0.0001	1	7.2500	103.0091	0.0	0.0	0.0	0.0002	0.0000	0.0000	0.0021	0.0165	0.0037	0.0285	0.0276	0.0273	0.0	0.0	0.0	0	0	0	0	0	0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
0.0204	1.1457	10000	0.2495	20.8519	0.8158	0.8223	0.8182	0.8489	0.8558	0.8514	0.8532	0.8583	0.8549	0.9474	0.9544	0.9500	0.9091	0.9167	0.9091	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.6364	0.6364	0.6364
0.0146	2.2915	20000	0.2638	19.4745	0.8330	0.8436	0.8375	0.8674	0.8788	0.8722	0.8717	0.8825	0.8763	0.9457	0.9559	0.9499	0.9091	0.9167	0.9167	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.4545	0.5	0.4762
0.0073	3.4372	30000	0.2853	18.3725	0.8473	0.8536	0.8497	0.8784	0.8855	0.8812	0.8825	0.8890	0.8850	0.9560	0.9615	0.9580	0.9167	0.9231	0.9231	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.5455	0.5455	0.5455
0.0041	4.5830	40000	0.3095	19.1354	0.8449	0.8556	0.8495	0.8788	0.8902	0.8836	0.8822	0.8931	0.8869	0.9487	0.9600	0.9535	0.9167	0.9167	0.9167	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.4	0.4444	0.4211
0.0035	5.7287	50000	0.3130	18.7752	0.8527	0.8654	0.8582	0.8839	0.8979	0.8900	0.8889	0.9018	0.8945	0.9456	0.9593	0.9516	0.9167	0.9231	0.9167	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
0.0031	6.8744	60000	0.3365	18.3513	0.8366	0.8516	0.8432	0.8664	0.8822	0.8734	0.8720	0.8859	0.8781	0.9446	0.9591	0.9509	0.9167	0.9231	0.9167	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
0.0013	8.0202	70000	0.3247	17.4401	0.8558	0.8673	0.8607	0.8863	0.8981	0.8912	0.8912	0.9018	0.8956	0.9551	0.9680	0.9607	0.9167	0.9231	0.9231	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.7	0.75	0.7368
0.0012	9.1659	80000	0.3487	18.4361	0.8485	0.8602	0.8535	0.8847	0.8972	0.8901	0.8890	0.8994	0.8933	0.9528	0.9641	0.9576	0.9167	0.9231	0.9231	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.6154	0.7273	0.6667
0.0041	10.3116	90000	0.3509	17.4401	0.8443	0.8585	0.8506	0.8753	0.8900	0.8818	0.8805	0.8931	0.8860	0.9515	0.9659	0.9578	0.9167	0.9231	0.9231	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.5	0.5556	0.5263
0.0006	11.4574	100000	0.3491	17.2706	0.8562	0.8671	0.8608	0.8862	0.8985	0.8914	0.8905	0.9018	0.8953	0.9555	0.9678	0.9608	0.9167	0.9231	0.9167	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.6923	0.7	0.7200
0.0001	12.6031	110000	0.3638	17.3342	0.8497	0.8605	0.8543	0.8779	0.8891	0.8826	0.8819	0.8914	0.8858	0.9556	0.9667	0.9602	0.9231	0.9231	0.9474	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.6667	0.6667	0.6667
0.002	13.7489	120000	0.3561	17.2494	0.8465	0.8556	0.8503	0.8769	0.8865	0.8809	0.8808	0.8893	0.8843	0.9563	0.9659	0.9602	0.9167	0.9231	0.9231	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.6923	0.7	0.7273
0.0012	14.8946	130000	0.3741	17.0587	0.8616	0.8756	0.8678	0.8932	0.9080	0.8998	0.8979	0.9111	0.9037	0.9561	0.9684	0.9615	0.9167	0.9231	0.9231	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.6923	0.6923	0.6923
0.0015	16.0403	140000	0.3661	17.2070	0.8474	0.8603	0.8530	0.8775	0.8915	0.8837	0.8831	0.8940	0.8878	0.9521	0.9640	0.9572	0.9167	0.9231	0.9231	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.6	0.6667	0.6316
0.0001	17.1861	150000	0.3762	17.4825	0.8490	0.8600	0.8538	0.8803	0.8921	0.8854	0.8861	0.8957	0.8901	0.9542	0.9628	0.9577	0.9167	0.9231	0.9167	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.6	0.6667	0.6316
0.0	18.3318	160000	0.3845	16.9951	0.8579	0.8683	0.8623	0.8874	0.8981	0.8919	0.8927	0.9013	0.8962	0.9525	0.9614	0.9561	0.9231	0.9231	0.9286	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.6667	0.7	0.6957
0.0	19.4775	170000	0.3866	16.7620	0.8581	0.8717	0.8641	0.8889	0.9034	0.8953	0.8935	0.9058	0.8988	0.9551	0.9667	0.9601	0.9231	0.9231	0.9333	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.6	0.6667	0.6316
0.0	20.6233	180000	0.3849	16.5713	0.8577	0.8688	0.8624	0.8862	0.8977	0.8911	0.8917	0.9009	0.8955	0.9552	0.9643	0.9589	0.9231	0.9231	0.9412	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.6154	0.7	0.6667
0.0	21.7690	190000	0.3930	16.5713	0.8611	0.8749	0.8671	0.8901	0.9044	0.8964	0.8959	0.9073	0.9007	0.9537	0.9644	0.9582	0.9231	0.9231	0.9286	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.6154	0.7	0.6667
0.0	22.9148	200000	0.3963	16.4230	0.8621	0.8754	0.8679	0.8906	0.9042	0.8966	0.8954	0.9066	0.9002	0.9532	0.9639	0.9577	0.9231	0.9231	0.9333	1.0	1.0	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.6154	0.7	0.6667

Framework versions

• Transformers 4.41.2
• Pytorch 2.2.1
• Datasets 2.20.0
• Tokenizers 0.19.1