swiftformer-xs-dmae-va-U5-42

This model is a fine-tuned version of MBZUAI/swiftformer-xs on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.0447
• Accuracy: 0.55

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 42

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
No log	0.9	7	1.3144	0.4833
1.3947	1.94	15	1.3154	0.4
1.3947	2.97	23	1.2849	0.4167
1.3608	4.0	31	1.2512	0.4667
1.3048	4.9	38	1.2340	0.55
1.3048	5.94	46	1.2118	0.5833
1.2456	6.97	54	1.2077	0.55
1.186	8.0	62	1.1672	0.5333
1.186	8.9	69	1.1565	0.6167
1.1218	9.94	77	1.1532	0.5833
1.0731	10.97	85	1.1304	0.5833
1.0731	12.0	93	1.1664	0.5167
1.0135	12.9	100	1.1222	0.55
0.9783	13.94	108	1.1404	0.5333
0.9783	14.97	116	1.1022	0.5833
0.9195	16.0	124	1.0996	0.55
0.9195	16.9	131	1.0715	0.6
0.9023	17.94	139	1.0779	0.5833
0.8575	18.97	147	1.0797	0.5667
0.8575	20.0	155	1.0508	0.5833
0.8519	20.9	162	1.0500	0.5833
0.8098	21.94	170	1.0212	0.5667
0.8098	22.97	178	1.0041	0.5833
0.8018	24.0	186	1.0197	0.5667
0.7709	24.9	193	1.0283	0.5333
0.7709	25.94	201	1.0303	0.55
0.7642	26.97	209	1.0100	0.5833
0.7322	28.0	217	1.0475	0.5333
0.7322	28.9	224	1.0667	0.55
0.7245	29.94	232	1.0743	0.55
0.7254	30.97	240	1.0416	0.5333
0.7254	32.0	248	1.0664	0.5667
0.7201	32.9	255	1.0393	0.55
0.7201	33.94	263	1.0284	0.55
0.6968	34.97	271	1.0420	0.55
0.7073	36.0	279	1.0396	0.5333
0.7073	36.9	286	1.0640	0.55
0.6891	37.94	294	1.0447	0.55

Framework versions

• Transformers 4.38.2
• Pytorch 2.2.1+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2