vihealthbert-w_unsup-SynPD

This model is a fine-tuned version of demdecuong/vihealthbert-base-word on the tmnam20/pretrained-vn-med-nli all dataset. It achieves the following results on the evaluation set:

• Loss: 1.5579
• Accuracy: 0.6891

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 32
• eval_batch_size: 16
• seed: 19144
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 10.0

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
5.8543	0.3446	2000	3.8967	0.3950
3.4544	0.6893	4000	2.8119	0.5306
2.8312	1.0339	6000	2.4040	0.5771
2.5914	1.3786	8000	2.6482	0.5350
2.5649	1.7232	10000	2.1335	0.6087
2.2749	2.0679	12000	1.9895	0.6282
2.1572	2.4125	14000	1.9313	0.6353
2.1009	2.7572	16000	1.8646	0.6429
2.0609	3.1018	18000	1.8572	0.6450
2.0885	3.4465	20000	1.9489	0.6285
1.9891	3.7911	22000	1.7700	0.6583
1.9368	4.1358	24000	1.7398	0.6609
1.9003	4.4804	26000	1.7165	0.6664
1.9058	4.8251	28000	1.7032	0.6670
1.859	5.1697	30000	1.6771	0.6718
1.8401	5.5144	32000	1.6652	0.6710
1.8181	5.8590	34000	1.6417	0.6754
1.8133	6.2037	36000	1.6431	0.6748
1.7888	6.5483	38000	1.6363	0.6755
1.7811	6.8930	40000	1.6205	0.6793
1.7481	7.2376	42000	1.6190	0.6807
1.7509	7.5823	44000	1.6142	0.6794
1.7517	7.9269	46000	1.5949	0.6819
1.7358	8.2716	48000	1.5909	0.6843
1.7287	8.6162	50000	1.5757	0.6851
1.7132	8.9609	52000	1.5671	0.6885
1.7042	9.3055	54000	1.5685	0.6867
1.7051	9.6502	56000	1.5609	0.6876
1.7051	9.9948	58000	1.5576	0.6883

Framework versions

• Transformers 4.40.2
• Pytorch 2.2.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1