swiftformer-xs-dmae-va-U5-42B

This model is a fine-tuned version of MBZUAI/swiftformer-xs on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.7864
• Accuracy: 0.75

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 128
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 42

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
No log	0.9	7	1.1573	0.6167
1.1018	1.94	15	1.1844	0.5667
1.1018	2.97	23	1.1120	0.55
0.9952	4.0	31	0.9493	0.6333
0.8441	4.9	38	0.9586	0.65
0.8441	5.94	46	1.0176	0.6167
0.7045	6.97	54	0.8787	0.6667
0.5583	8.0	62	1.3863	0.5333
0.5583	8.9	69	0.9727	0.6667
0.457	9.94	77	1.0685	0.6167
0.3699	10.97	85	1.1291	0.6667
0.3699	12.0	93	0.9419	0.6667
0.3095	12.9	100	0.8806	0.6333
0.2544	13.94	108	0.8804	0.7167
0.2544	14.97	116	0.8399	0.65
0.2175	16.0	124	0.8857	0.6333
0.2175	16.9	131	0.9544	0.6667
0.2152	17.94	139	1.0697	0.6833
0.2011	18.97	147	1.0718	0.6667
0.2011	20.0	155	1.0276	0.6667
0.182	20.9	162	0.8614	0.6833
0.1591	21.94	170	0.7872	0.6833
0.1591	22.97	178	1.0134	0.65
0.1557	24.0	186	0.9191	0.65
0.1476	24.9	193	0.8978	0.65
0.1476	25.94	201	0.8872	0.7
0.1587	26.97	209	1.0397	0.7
0.1082	28.0	217	0.9320	0.6833
0.1082	28.9	224	0.8857	0.6667
0.1345	29.94	232	0.8767	0.6667
0.1308	30.97	240	0.8945	0.6833
0.1308	32.0	248	0.9545	0.6833
0.1348	32.9	255	0.9330	0.7
0.1348	33.94	263	0.9131	0.7167
0.1162	34.97	271	0.8424	0.7167
0.1134	36.0	279	0.8275	0.6833
0.1134	36.9	286	0.8106	0.7333
0.1108	37.94	294	0.7864	0.75

Framework versions

• Transformers 4.38.2
• Pytorch 2.2.1+cu121
• Datasets 2.18.0
• Tokenizers 0.15.2